[00:32:46] 10DBA, 10Community-Tech-Sprint, 10MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), 10MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), 10Patch-For-Review: Issue with maintenance script: SELECTing revisions with high rev_id is pai... - https://phabricator.wikimedia.org/T175962#3630601 [04:27:48] 10DBA, 10Operations, 10Patch-For-Review: db2047 got rebooted - https://phabricator.wikimedia.org/T176573#3630077 (10Marostegui) I can see HW logs and it found this: ``` hpiLO-> show record14 status=0 status_tag=COMMAND COMPLETED Mon Sep 25 04:27:07 2017 /system1/log1/record14 Targets... [04:28:54] 10DBA, 10Operations, 10Patch-For-Review: db2047 got rebooted - https://phabricator.wikimedia.org/T176573#3630681 (10Marostegui) >>! In T176573#3630142, @Volans wrote: > @jcrespo interesting, I guess the documentation in https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/HP_DL3N0#Show_system_... [04:34:29] 10DBA, 10Operations, 10Patch-For-Review: db2047 got rebooted - https://phabricator.wikimedia.org/T176573#3630682 (10Marostegui) I have checked other hosts on that same rack (C6) and there are no warnings on ILO or anything related. @Papaul can you visually check the rack to see if there is any temperature wa... [04:37:38] 10DBA, 10Operations, 10Patch-For-Review: decommission db1018 - https://phabricator.wikimedia.org/T176215#3630688 (10Marostegui) [04:43:15] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3630692 (10Marostegui) @MarkTraceur any update on this? Were you able to try to arrange a read-only time for s4? [05:00:59] 10DBA, 10Operations, 10ops-eqiad: Decommission db1049 - https://phabricator.wikimedia.org/T175264#3630693 (10Marostegui) a:03Cmjohnson As per Jaime comment, this is now ready to be handed over Chris. Thanks! [05:19:17] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3630697 (10Marostegui) Thanks @Reedy! If we want to do a bit of clean... [05:32:12] 10DBA, 10Wikidata: Provision a separate DB shard for wbc_entity_usage - https://phabricator.wikimedia.org/T176277#3630712 (10Marostegui) >>! In T176277#3619796, @hoo wrote: > Additional question: (How) Would this work for labs? There should not be too many issues for this to apply to the new labs servers, as... [07:42:04] 10DBA, 10Operations, 10Patch-For-Review: decommission db1018 - https://phabricator.wikimedia.org/T176215#3630783 (10Marostegui) [07:42:37] 10DBA, 10Operations, 10ops-eqiad: decommission db1018 - https://phabricator.wikimedia.org/T176215#3617573 (10Marostegui) a:03Cmjohnson [07:49:49] 10DBA, 10Operations, 10Patch-For-Review: decommission db1036 - https://phabricator.wikimedia.org/T176311#3630800 (10Marostegui) [07:49:57] 10DBA, 10Operations, 10Patch-For-Review: decommission db1036 - https://phabricator.wikimedia.org/T176311#3620829 (10Marostegui) [07:51:27] 10DBA, 10Operations, 10Patch-For-Review: decommission db1036 - https://phabricator.wikimedia.org/T176311#3620829 (10Marostegui) [09:09:31] 10DBA, 10Operations, 10Patch-For-Review: decommission db1036 - https://phabricator.wikimedia.org/T176311#3630928 (10Marostegui) a:03Marostegui db1036 is now fully depooled and db1101 is serving as a special slave. Let's give it a couple of days and then proceed to get rid of db1036 from all the config file... [09:12:40] check for slowdowns on tendril/errors on pooling [09:12:49] I did the partitioning (and improved it) [09:12:58] thanks :) [09:13:02] but you never now [09:13:06] *know [09:13:10] if I did something wrong [09:13:33] I am doubting on my work, not yours, to be clear [09:13:38] haha [09:13:40] no worries :) [09:13:51] I am doing it :) [10:19:43] 10DBA, 10Community-Tech-Sprint, 10MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), 10MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), 10Patch-For-Review: Issue with maintenance script: SELECTing revisions with high rev_id is pai... - https://phabricator.wikimedia.org/T175962#3631131 [10:46:47] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, 10MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), 10Schema-change: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3631191 (10MarcoAurelio) eswiki, eswikibooks and metawiki tables can g... [10:49:52] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3631212 (10jcrespo) [10:51:54] 10DBA, 10MediaWiki-Platform-Team, 10Structured-Data-Commons, 10Wikidata: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#3631228 (10jcrespo) [10:53:54] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3631246 (10Marostegui) >>! In T172207#3631191, @MarcoAurelio wrote: > eswiki, eswikibooks and metawiki tables c... [11:06:18] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10MediaWiki-extensions-UserMerge, and 2 others: flaggedrevs.fr_user is unindexed - https://phabricator.wikimedia.org/T172207#3631283 (10Marostegui) However, there is nothing blocking us now from deploying the index on the wikis where we... [11:28:04] 10DBA, 10Cloud-Services: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3631300 (10MarcoAurelio) @Marostegui Wiki creation is happening now. [11:33:37] 10DBA, 10Cloud-Services: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3631348 (10Marostegui) >>! In T173027#3631300, @MarcoAurelio wrote: > @Marostegui Wiki creation is happening now. I can see it is already created on the master. I will check the triggers and... [11:57:18] 10DBA, 10Cloud-Services: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3631402 (10Marostegui) I have ran redact_sanitarium on db1095 and I can see that the user, revision and recentchanges tables have been sanitized correctly. I am running the check_private data... [12:07:08] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2047 got rebooted - https://phabricator.wikimedia.org/T176573#3631438 (10Marostegui) [12:24:15] 10DBA: Run pt-table-checksum on s3 - https://phabricator.wikimedia.org/T164488#3631550 (10Marostegui) [12:24:49] 10DBA, 10Operations: Decommission db1015, db1035, db1044 and db1038 - https://phabricator.wikimedia.org/T148078#3631553 (10Marostegui) db1015 was scheduled for decommission: T173570 [12:25:06] 10DBA, 10Operations: Decommission db1015, db1035, db1044 and db1038 - https://phabricator.wikimedia.org/T148078#3631555 (10Marostegui) [12:30:28] 10DBA: Decommission database hosts < db2030 (tracking) - https://phabricator.wikimedia.org/T176243#3631582 (10Marostegui) p:05Triage>03Normal [13:03:29] hi, I'm going through existing wikitech docs to update salt docs and either remove them or change them to cumin commands. there's also https://wikitech.wikimedia.org/wiki/MariaDB#Salt. but for this I'm not familiar with the replacement, could any of the DBAs please update this? [13:12:47] 10DBA, 10Cloud-Services: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3631740 (10Marostegui) The report finished on labsdb1009 and I can see the new users are being sanitized correctly on all hosts. I will wait for the report to finish on labsdb1001 before handl... [13:15:44] 10DBA, 10Community-Tech-Sprint, 10MW-1.30-release-notes (WMF-deploy-2017-09-19 (1.30.0-wmf.19)), 10MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), 10Patch-For-Review: Issue with maintenance script: SELECTing revisions with high rev_id is pai... - https://phabricator.wikimedia.org/T175962#3631743 [14:55:30] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3632148 (10MarkTraceur) Oh, I'm sorry, I missed some updates here (lots of noise on this ticket), and misunderstood that I needed to go through some process... [15:09:46] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2047 got rebooted - https://phabricator.wikimedia.org/T176573#3632248 (10Papaul) The temperature in C6 is about 106 F and the ILO log shows Critical Temperature Threshold Exceeded on 9-24 at 11:30 pm (see attachment) {F9800902} [15:28:11] 10DBA, 10MediaWiki-Platform-Team, 10Structured-Data-Commons, 10Wikidata: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#3632325 (10daniel) Well, it actually is blocked on the schema change. We can't fully deploy MCR without it. But I suppose the tag is for situations where something i... [15:28:26] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2047 got rebooted - https://phabricator.wikimedia.org/T176573#3632327 (10Marostegui) >>! In T176573#3632248, @Papaul wrote: > The temperature in C6 is about 106 F and the ILO log shows Critical Temperature Threshold Exceeded on 9-24 at 11:30 pm (see a... [15:32:16] 10DBA, 10Cloud-Services: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3632349 (10Marostegui) All good on labsdb1001 as well. #cloud-services-team feel free to create the views Thanks! [15:35:12] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3632359 (10Marostegui) >>! In T168661#3632148, @MarkTraceur wrote: > Oh, I'm sorry, I missed some updates here (lots of noise on this ticket), and misunderst... [15:47:50] 10DBA, 10MediaWiki-Platform-Team, 10Structured-Data-Commons, 10Wikidata: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#3549134 (10Anomie) #blocked-on-schema-change is used when the schema change is ready to go and further progress is blocked on the change actually being made on WMF's... [15:52:04] 10DBA, 10MediaWiki-Platform-Team, 10Structured-Data-Commons, 10Wikidata: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044#3632402 (10jcrespo) @Anomie is right, adding it now would only spam us and deplay actually blocked changes. Add the DBA to tell us we will be involved in the future,... [16:21:46] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Decommission db1023 - https://phabricator.wikimedia.org/T166486#3297505 (10Dzahn) @cmjohnson db1022 and db1023 and their mgmt interfaces are alerting as down in Icinga. Did you shut them down? Looks like they still need to be removed from puppet. [16:24:56] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Decommission db1023 - https://phabricator.wikimedia.org/T166486#3632505 (10Cmjohnson) @dzahn yes they're down... i think these pre-dated the mgmt interface icinga check [16:25:54] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: Decommission db1023 - https://phabricator.wikimedia.org/T166486#3632506 (10Dzahn) @Cmjohnson Ah, thanks! It's both, the servers and mgmt though. https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&style=hostdetail&hoststatustypes=4&hostprops... [16:58:16] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2047 got rebooted - https://phabricator.wikimedia.org/T176573#3632627 (10Marostegui) Looking at the PDU temperature graphs, I cannot see anything weird there, so it might have been just a punctual thing with this host. [17:23:34] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Apply schema change to add 3D filetype for STL files - https://phabricator.wikimedia.org/T168661#3632757 (10greg) What @Marostegui said, the only really more complicated part is the announce/delay so that people are aware. I'm not sure @klove is the rig... [20:24:08] 10DBA, 10Cloud-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3633325 (10MarcoAurelio) [21:10:42] 10DBA, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, 10Patch-For-Review, and 2 others: Usage tracking: record which statement group is used - https://phabricator.wikimedia.org/T151717#3633466 (10hoo) Populating `elwiki` will be done in the next few hours, I'll then post updated stats. In order to kee... [21:44:17] 10DBA, 10Cloud-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for hi.wikivoyage - https://phabricator.wikimedia.org/T173027#3633590 (10bd808) [23:32:58] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Create and announce timeline for shutting down labsdb100[13] - https://phabricator.wikimedia.org/T175086#3633776 (10bd808) @jcrespo has suggested Wednesday 2017-12-13 as the target shutdown date for the servers. We also need to choose a date sometime in... [23:46:14] 10DBA, 10Data-Services, 10Goal, 10cloud-services-team (FY2017-18): Decommission labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T142807#3633796 (10bd808) [23:47:17] 10DBA, 10Data-Services, 10User-bd808, 10cloud-services-team (Kanban): Create and announce timeline for shutting down labsdb100[13] - https://phabricator.wikimedia.org/T175086#3633797 (10bd808) a:03bd808