[05:30:01] 10DBA, 10Labs, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3376801 (10Marostegui) Hi As per: T168782 this looks like a public wiki but with some restrictions, what about its replication to labs hosts? Does it need to happen as a normal publi... [05:32:33] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Green Cardamom → GreenC: supervision needed - https://phabricator.wikimedia.org/T168776#3377466 (10Marostegui) p:05Triage>03Normal Looks like most of the edits are on enwiki as per: https://meta.wikimedia.org/wiki/Special:CentralAuth/Green_... [06:22:42] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s7 - https://phabricator.wikimedia.org/T166208#3377553 (10Marostegui) [06:40:47] 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3377565 (10Marostegui) [06:42:06] 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#3377567 (10Marostegui) [06:47:48] 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#3377584 (10Marostegui) [06:49:51] 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#3377586 (10Marostegui) [06:51:48] 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#3377589 (10Marostegui) [06:55:54] 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#3377593 (10Marostegui) [06:58:19] 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#3377605 (10Marostegui) [06:58:34] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3377608 (10Marostegui) [06:58:36] 10DBA, 10Operations: Drop wikilove_image_log table from Wikimedia wikis - https://phabricator.wikimedia.org/T127219#2036092 (10Marostegui) 05Open>03Resolved This is all done [07:09:09] 10DBA, 10Labs, 10Labs-Infrastructure, 10Tracking: LabsDB replica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#3377617 (10Marostegui) [07:09:12] 10DBA, 10Labs, 10Patch-For-Review: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#3377614 (10Marostegui) 05Open>03Resolved a:03Marostegui I have cleaned up the views, the only pending thing would be to merge: https://gerr... [07:14:16] 10DBA: Point labsdb1001 and labsdb1003 to db1095 - https://phabricator.wikimedia.org/T166546#3377619 (10Marostegui) a:03Marostegui [07:50:17] 10DBA: Truncate 'searchindex' table on all wikis - https://phabricator.wikimedia.org/T167978#3377679 (10Marostegui) a:03Marostegui The current situation is: s1 -> the table isn't there s2 -> the table isn't there s3 table present on 682 wikis, all of them are empty apart from: ``` azwiki 1 cywiki 1 dawiki 4... [08:10:17] 10DBA: Point labsdb1001 and labsdb1003 to db1095 - https://phabricator.wikimedia.org/T166546#3377748 (10Marostegui) I just realised that changing the master from labsdb1001 and labsdb1003 for the above threads, need to be done all at the same time, otherwise it will break Why? Because db1095 has a binlog, which... [08:19:02] 2 weeks later, I still do not know what is going on with pc* hosts [08:19:30] in which way? [08:19:34] disk usage [08:20:42] oh shit, still growing a lot [08:53:44] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3377904 (10Marostegui) [08:53:45] 10DBA: Truncate 'searchindex' table on all wikis - https://phabricator.wikimedia.org/T167978#3377901 (10Marostegui) 05Open>03Resolved Table has been truncated where it existed. [08:55:36] 10DBA: Truncate 'searchindex' table on all wikis - https://phabricator.wikimedia.org/T167978#3377906 (10jcrespo) I see it missing from the hosts you mentioned above. [08:56:09] what do you mean? [08:56:41] https://phabricator.wikimedia.org/T167978#3353966 [08:56:50] Aaaah [08:56:54] Right :) [08:57:01] Sure, I can create them :) [08:57:04] it is ok if it is a different ticket [08:57:09] Nah, not worth! [08:57:11] but I do not want to give the impression [08:57:17] it is fixed [08:57:48] "if we're looking for parity with core we should probably recreate the empty tables on those?" [08:57:56] yeah [08:57:58] it was on the description [08:58:14] Yes yes, you are right [08:58:22] I will quickly create them if don't exist [08:58:43] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3377910 (10Marostegui) [08:58:45] 10DBA: Truncate 'searchindex' table on all wikis - https://phabricator.wikimedia.org/T167978#3377909 (10Marostegui) 05Resolved>03Open [08:58:46] having the same tables everywhere should be one of our goals [08:58:58] to avoid snowflakes [08:59:06] yes, totally agree :) [09:06:06] 10DBA: Truncate 'searchindex' table on all wikis - https://phabricator.wikimedia.org/T167978#3377923 (10Marostegui) Created on the wikis that it was missing from [09:07:35] 10DBA: Truncate 'searchindex' table on all wikis - https://phabricator.wikimedia.org/T167978#3377925 (10Marostegui) 05Open>03Resolved [09:07:39] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3377926 (10Marostegui) [12:46:18] 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3378669 (10Marostegui) I have done the alter on s2 (codfw master and let it replicate) and looks like a table rebuilt is triggered: ``` stage: 1 of 2 'copy to tmp table' 2.24% o... [12:47:47] 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3378672 (10Marostegui) [12:54:10] 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3378729 (10jcrespo) @jcrespo yes, but please keep track of altered and unaltered tables somewhere easy to find. [12:54:53] 10DBA, 10Labs, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3378730 (10Urbanecm) Hi @Marostegui, I think the wiki should be replicated as other public wiki. In fact all other fishbowl (publicly readable, editing restricted) wikis are replicated. [12:55:37] 10DBA, 10Labs, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3378732 (10Marostegui) >>! In T168788#3378730, @Urbanecm wrote: > Hi @Marostegui, I think the wiki should be replicated as other public wiki. In fact all other fishbowl (publicly reada... [12:56:03] 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3378734 (10Marostegui) >>! In T168661#3378729, @jcrespo wrote: > @jcrespo yes, but please keep track of altered and unaltered tables somewhere easy to find. Yes - I will keep tra... [13:02:38] 10DBA, 10Labs, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3378752 (10Urbanecm) There are listed at https://github.com/wikimedia/operations-mediawiki-config/blob/master/dblists/fishbowl.dblist. I can do use _p at toollabs for all of th... [13:13:03] 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3378770 (10jcrespo) [13:14:15] 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3378782 (10Marostegui) This looks like a bug to me so I will report it: db2062 (10.1): Forcing `ALGORITHM=INPLACE` ``` root@db2062.codfw.wmnet[enwiki]> select @@version; +-------... [13:14:36] 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3378783 (10Marostegui) [13:32:13] 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3378850 (10Marostegui) Let's see what MariaDB says: https://jira.mariadb.org/browse/MDEV-13175 [13:39:11] 10DBA, 10Labs, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3378877 (10Marostegui) Thanks @Urbanecm. I have checked them and they are in labs indeed - not that I didn't trust you, I just wanted to double check, better be safe than leaking some... [13:40:56] 10DBA, 10Patch-For-Review: dbstore2001: s5 thread isn't able to catch up with the master - https://phabricator.wikimedia.org/T168354#3378881 (10Marostegui) s5 is now replicating again in dbstore2001 after a few days getting it imported. Right now, all the shards are a few days delayed due to the increase in lo... [14:13:42] 10DBA, 10Labs, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3378981 (10Urbanecm) Thank you for checking! Unfortunately I can't ping you when the tables are in prod as I can't create it. I can only prepare relevant configuration patches (as you... [14:49:17] 10Blocked-on-schema-change, 10DBA: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3379134 (10Marostegui) [15:27:03] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s7 - https://phabricator.wikimedia.org/T166208#3379282 (10Marostegui) [17:30:47] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Green Cardamom → GreenC: supervision needed - https://phabricator.wikimedia.org/T168776#3379913 (10alanajjar) @Marostegui can we start now? [17:37:10] 10DBA, 10ArchCom-RfC, 10RESTBase-API, 10Reading List Service, and 4 others: RfC: Reading List service - https://phabricator.wikimedia.org/T164990#3379944 (10daniel) This RFC has entered the //Last Call// period. If no new and pertinent concerns remain unaddressed by July 5, this RFC will be approved for im... [20:27:33] 10DBA, 10Operations, 10Wikimedia-Site-requests: Global rename of Green Cardamom → GreenC: supervision needed - https://phabricator.wikimedia.org/T168776#3376531 (10Luke081515) @alanajjar I guess it would make more sense to ping him at IRC, so if there is a problem, you can communicate much faster. [20:29:21] 10DBA, 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests: Reopen Wikinews Dutch - https://phabricator.wikimedia.org/T168764#3380415 (10Luke081515) Adding #DBA to get an opinion from one of them.