[00:28:28] <wikibugs>	 10DBA, 10Analytics, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3530850 (10MZMcBride) This task is potentially of interest to me. I previously ran a dumb report with...
[00:53:32] <wikibugs>	 10DBA, 10Analytics, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3739451 (10bd808) >>! In T173511#3739381, @MZMcBride wrote: > If this task were implemented, under "Pr...
[06:16:40] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3739752 (10Marostegui) s2 codfw is done. No pasting here the whole check, but this reveals tha...
[06:16:52] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3739753 (10Marostegui)
[08:47:31] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Drop now redundant indexes from pagelinks and templatelinks - https://phabricator.wikimedia.org/T174509#3739954 (10Marostegui)
[08:53:04] <elukey>	 marostegui: first run of eventlogging_cleaner for data up to middle of 2016 completed \o/
[08:53:27] <elukey>	 we never even got close to this result before with weeks of tests
[08:53:33] <marostegui>	 How cool!!
[08:53:47] <marostegui>	 I am so happy we will be able to get rid of those crappy old hosts
[09:13:36] <wikibugs>	 10DBA, 10Patch-For-Review: Decommission db1038 - https://phabricator.wikimedia.org/T177911#3740008 (10Marostegui)
[09:15:11] <wikibugs>	 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1038 - https://phabricator.wikimedia.org/T177911#3740010 (10Marostegui) This host is fully ready to be decommissioned by @Cmjohnson
[09:15:24] <wikibugs>	 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1038 - https://phabricator.wikimedia.org/T177911#3740015 (10Marostegui) a:05Marostegui>03Cmjohnson
[09:16:17] <wikibugs>	 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1050 - https://phabricator.wikimedia.org/T178162#3740016 (10Marostegui)
[09:16:30] <wikibugs>	 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: decommission db1036 - https://phabricator.wikimedia.org/T176311#3740017 (10Marostegui)
[09:16:45] <wikibugs>	 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1035 - https://phabricator.wikimedia.org/T176931#3740019 (10Marostegui)
[09:17:04] <wikibugs>	 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1049 - https://phabricator.wikimedia.org/T175264#3740020 (10Marostegui)
[09:17:18] <wikibugs>	 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1037 - https://phabricator.wikimedia.org/T174902#3740021 (10Marostegui)
[09:17:32] <wikibugs>	 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1033 and db1028 - https://phabricator.wikimedia.org/T174076#3740022 (10Marostegui)
[09:17:48] <wikibugs>	 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1026 - https://phabricator.wikimedia.org/T174763#3740023 (10Marostegui)
[09:18:03] <wikibugs>	 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1041 - https://phabricator.wikimedia.org/T173915#3740024 (10Marostegui)
[09:41:33] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3740048 (10Marostegui)
[10:00:32] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3740403 (10Marostegui)
[10:02:28] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3740413 (10Marostegui) I have sanitized sanitarium and checked that itself and labs got sanitized correctly. No private data was re...
[10:02:39] <wikibugs>	 10DBA, 10Cloud-Services, 10User-Jayprakash12345, 10cloud-services-team (Kanban): Prepare and check storage layer for hif.wiktionary - https://phabricator.wikimedia.org/T173647#3740417 (10Marostegui) p:05Low>03Normal
[10:03:47] <elukey>	 qq - it came up after a chat with my team that usually researchers create tables on a schema called 'staging'
[10:04:27] <marostegui>	 elukey: yep, i asked you if we wanted staging db there, and you said no, only log, that is why i didn't migrate it :)
[10:04:41] <elukey>	 yeah I didn't know it at the time :)
[10:05:12] <elukey>	 it is a mix of data from wikis and log afaik, so not sure where it would be best to copy
[10:05:41] <elukey>	 I am thinking to have a new staging schema on db1108 and use
[10:05:42] <elukey>	 GRANT ALTER, CREATE, CREATE TEMPORARY TABLES, DELETE, DROP, INDEX, INSERT, SELECT, UPDATE ON `staging`.* TO '%'@'%';
[10:05:58] <elukey>	 (that is one of the grants on db1047 now)
[10:06:03] <marostegui>	 sure
[10:06:04] <elukey>	 too broad?
[10:06:06] <marostegui>	 that also works
[10:06:21] <marostegui>	 well, they can basically do whatever they want on that schema :)
[10:06:24] <marostegui>	 with those grants XD
[10:06:30] <marostegui>	 but only on that schema
[10:06:53] <elukey>	 oh yes that makes sense, it should be a scratch pad
[10:07:00] <marostegui>	 then, that is fine :)
[10:07:24] <elukey>	 so CREATE DATABASE 'staging' and then the grant
[10:07:40] <marostegui>	 yep!
[10:07:51] <marostegui>	 with the password and everything
[10:07:57] <elukey>	 would it be possible to copy the staging db from db1047 to dbstore1002 changing the schema name?
[10:08:08] <marostegui>	 what do you mean
[10:08:29] <marostegui>	 there is already a staging db on dbstore1002
[10:08:48] <elukey>	 yeah but the data in there is completely different 
[10:08:52] <marostegui>	 ah right
[10:09:12] <marostegui>	 so you'd want a staging2, for instance
[10:09:24] <elukey>	 yes exactly, or staging_db1047
[10:09:37] <marostegui>	 we could do that, but how big it is and how much space we have in dbstore1002?
[10:10:03] <elukey>	 need to check, but it would be good to do after we drop the log db in there
[10:10:34] <marostegui>	 yeah, it is easy to do
[10:10:39] <marostegui>	 as long as we have space
[10:10:52] <marostegui>	 i assume it is not too big anyways, no?
[10:10:58] <elukey>	 checking
[10:12:27] <elukey>	 biggest table is around 42GB 
[10:12:48] <elukey>	 checking the engine
[10:13:07] <elukey>	 Aria
[10:13:15] <marostegui>	 haha
[10:13:34] <marostegui>	 we should migrate it to innodb if possible
[10:14:12] <elukey>	 (I checked with show create table, not sure if enough)
[10:14:19] <marostegui>	 yeah, that is enough
[10:14:29] <elukey>	 they seems all Aria, or at least most of them
[10:14:45] <elukey>	 can we move them to innodb while copying the db ?
[10:14:59] <elukey>	 or is it a manual work to do before?
[10:15:07] <marostegui>	 i would convert them before, and then move them
[10:15:15] <elukey>	 all right
[10:15:20] * elukey takes notes
[10:15:43] <marostegui>	 i assume the only reason they are aria is because it was the default and they didn't specify engine=innodb or tokudb upon creation right?
[10:15:55] <marostegui>	 not like they are using insert delayed or specific aria (myisam) stuff
[10:17:33] <elukey>	 I am preparing a section for that database in the email, will ask people to chime in the task and add their requirements
[10:17:41] <marostegui>	 sounds good!
[10:17:45] <marostegui>	 cc me if you like
[10:17:58] <marostegui>	 cumin
[10:18:00] <elukey>	 it will go to engineering@ so you'll see it :)
[10:18:06] <marostegui>	 ah right!
[10:18:18] <elukey>	 so going to create the database on db1108
[10:18:25] <marostegui>	 ok!
[10:18:35] <marostegui>	 as a good practice: create database if not exists
[10:18:41] * elukey notes
[10:18:42] <marostegui>	 now it doesn't matter, because there is no replicaiton involved
[10:18:47] <marostegui>	 but just so you know :)
[10:51:46] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3740545 (10Marostegui)
[10:57:55] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3740578 (10Marostegui)
[13:24:50] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3740899 (10Marostegui) s7 codfw is done. ``` for i in `cat /home/marostegui/git/mediawiki-conf...
[13:25:00] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3740900 (10Marostegui)
[13:50:59] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3740936 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` db1105.eqiad.wmnet ``` The log can be found in `/var/log/wmf-auto-reimage/2017110...
[13:57:46] <wikibugs>	 10DBA, 10Wikidata: Migrate wb_items_per_site to using prefixed entity IDs instead of numeric IDs - https://phabricator.wikimedia.org/T114904#3740943 (10daniel) We should also consider a solution that will only put the full ids on labs. We don't actually need it in production. Could be a separate table, kept up...
[14:08:53] <wikibugs>	 10DBA, 10Patch-For-Review: Support multi-instance on core hosts - https://phabricator.wikimedia.org/T178359#3740991 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1105.eqiad.wmnet'] ```  and were **ALL** successful.
[14:09:03] <marostegui>	 \o/
[14:38:39] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3741233 (10Marostegui)
[16:00:19] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1059 - https://phabricator.wikimedia.org/T179727#3734260 (10RobH) Please note this is out of warranty.  Disk failed:  Enclosure Device ID: 32 Slot Number: 0 Drive's position: DiskGroup: 0, Span: 0, Arm: 0 Enclosure position: 1 Device Id: 0 WWN: 5000C5005E6...