[00:08:36] 10DBA, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, 10Flow, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2695552 (10Dereckson) [00:09:13] 10DBA, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, 10Flow, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2054159 (10Dereckson) Flow tables have been created. [06:20:28] 10DBA, 06Operations, 10ops-eqiad: db1055: degraded array - https://phabricator.wikimedia.org/T147172#2695959 (10Marostegui) Hi, The disk is still being rebuilt: ``` root@db1055:~# megacli -PDRbld -ShowProg -PhysDrv [32:0] -aALL Rebuild Progress on Device at Enclosure 32, Slot 0 Completed 76% in 952 Minut... [07:51:16] 10DBA, 06Operations: Drop database table "email_capture" from Wikimedia wikis - https://phabricator.wikimedia.org/T57676#2696041 (10Marostegui) Table in S1 has been deleted. [08:34:31] 10DBA, 06Operations: Drop database table "email_capture" from Wikimedia wikis - https://phabricator.wikimedia.org/T57676#2696076 (10Marostegui) Table in S3 has been deleted. I believe this ticket can be closed. Looks like email_capture isn't present in any other shard. [08:50:50] 10DBA: Unify commonswiki.revision - https://phabricator.wikimedia.org/T147305#2696085 (10Marostegui) Looks like the master is getting some SELECTs (we will fill a bug report for this later). But we are going to get it ALTEREd so we can avoid that small degradation. To get the master altered I would run: #./sof... [09:25:58] 10DBA, 06Operations, 13Patch-For-Review: Investigate db1082 crash - https://phabricator.wikimedia.org/T145533#2633433 (10Marostegui) For now I have restored its original value until we agreed on when we can upgrade it. So far it has been behaving fine since it crashed around a month ago. [09:52:27] marostegui: I dropped the unpacked firmware on db1082 btw, in my home [09:52:45] godog: Ah thanks I was running cpio :) [09:53:14] heheh good old rpm2cpio, fun times [09:53:31] That and alien :p [09:53:36] How did you unpack yourself? [09:56:42] eheheh alien! [09:57:12] haha classic one! [09:57:56] godog: So it would be a matter of ./hpsetup now? (don't think will do it now though) [10:02:30] marostegui: yeah [10:02:42] godog: Cool - thank you! :) [10:03:23] np! easy enough, at least hp supports linux, I was already expecting a dos-mode .exe to be ran from a floppy [10:03:33] XDDDDDDDD [10:03:43] Wouldn't be the first time :p [10:22:26] 10DBA: Fatal error: Call to protected method Database::makeSelectOptions - https://phabricator.wikimedia.org/T147550#2696181 (10mwjames) [10:27:36] 10DBA: Fatal error: Call to protected method Database::makeSelectOptions - https://phabricator.wikimedia.org/T147550#2696197 (10mwjames) [10:35:55] 10DBA: Fatal error: Call to protected method Database::makeSelectOptions - https://phabricator.wikimedia.org/T147550#2696181 (10Marostegui) Hello, Thanks for the report - however I don't think the tag DBA is appropriate for this report as we are not responsible for that part of the infrastructure. I would su... [10:51:34] 10DBA, 06Operations, 10ops-eqiad: db1055: degraded array - https://phabricator.wikimedia.org/T147172#2696224 (10Marostegui) The rebuilt process finished successfully this time - so it was indeed the disk as you said: ``` Device Present ================ Virtual Drives : 1... [10:51:46] 10DBA, 06Operations, 10ops-eqiad: db1055: degraded array - https://phabricator.wikimedia.org/T147172#2696225 (10Marostegui) 05Open>03Resolved [10:59:40] godog: Did you have any problems when upgrading the firmware? ie: boxes not booting up? I am not sure if I want to do it with db1082 today, and certainly not tomorrow just in case it doesn't come back [11:00:59] marostegui: no problems so far, it was on fairly new hp machines though, not sure about db1082 [11:01:16] I agree there's a chance of it not coming back up, so possibly next week [11:01:20] godog: It is one of the new ones (I think) [11:02:09] ah ok, then yeah it should just work, at least on ms-be machines I had no problems [11:02:35] good to know - thanks [11:15:26] 10DBA: hitcounter and _counter tables are on the cluster but were deleted/unsused? - https://phabricator.wikimedia.org/T132837#2696256 (10Marostegui) Dropped `hitcounter` and`_counters` tables from S1 enwiki [11:24:04] 10DBA, 06Operations, 10ops-eqiad: db1065: Degraded RAID - https://phabricator.wikimedia.org/T147396#2691693 (10Cmjohnson) Disk has been requested through the Dell portal. Confirmed: Request 937313705 was successfully submitted. Your service request has been successfully created and will be reviewed by our... [11:26:01] cmjohnson1: Thanks - do we have NBD? Just curious [11:26:29] NBD? [11:26:39] Next Business Day [11:26:47] For the hardware stuff, like disks and that [11:26:55] oh..yes, we do [11:27:16] Great :) [11:41:01] 10DBA: hitcounter and _counter tables are on the cluster but were deleted/unsused? - https://phabricator.wikimedia.org/T132837#2696275 (10Marostegui) Dropped hitcounter and`_counters` tables from S2: bgwiki bgwiktionary cswiki enwikiquote enwiktionary eowiki fiwiki idwiki itwiki nlwiki nowiki plwiki ptwiki svwik... [11:57:00] 10DBA: hitcounter and _counter tables are on the cluster but were deleted/unsused? - https://phabricator.wikimedia.org/T132837#2696293 (10Marostegui) dbstore2002 (as expected) failed too [13:35:44] 10DBA, 06Operations, 10ops-eqiad: db1065: Degraded RAID - https://phabricator.wikimedia.org/T147396#2696401 (10Cmjohnson) The Reference Dispatch Number is: 321760647 Your part dispatch will be delivered to the following location: Wikimedia c/o of Equinix, 21721 Filigree Ct. Cage 61130 Ashburn, VA 20147 [13:38:56] 10DBA, 06Operations, 10ops-eqiad: db1065: Degraded RAID - https://phabricator.wikimedia.org/T147396#2696423 (10Marostegui) Awesome - thanks for the heads up [13:42:10] 10DBA: Unify commonswiki.revision - https://phabricator.wikimedia.org/T147305#2696438 (10Marostegui) This will be run tomorrow morning CEST [14:15:28] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), 13Patch-For-Review: MySQL chooses poor query plan for link counting query - https://phabricator.wikimedia.org/T143932#2696726 (10TJones) [14:22:36] I am going to deploy https://gerrit.wikimedia.org/r/314465 [14:23:02] and then restart db1069:3313 [14:23:07] sounds good [14:23:17] just FYI [14:23:21] May I ask why it needs to be restarted? [14:23:24] No need to answer now [14:23:25] :) [14:23:51] reload the replication filter [14:24:03] we do not trust puppet to touch mysql [14:24:11] ah :) [14:24:18] forgot about the filters [14:24:19] :) [14:54:58] I have applied a filter to all servers, just in case [14:54:58] https://tools.wmflabs.org/replag/ should go down now [15:07:54] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: Prepare storage layer for olo.wikipedia - https://phabricator.wikimedia.org/T147302#2696868 (10jcrespo) a:03jcrespo Claimed, but will be done together with @Marostegui for demonstration purposes. [15:09:24] 10DBA, 06Labs, 10Labs-Infrastructure, 06Operations: Prepare and check production and labs-side filtering for olowiki - https://phabricator.wikimedia.org/T147302#2696878 (10jcrespo) [17:38:27] marostegui: did you possibly have any time to add some more canary/test data to labsdb1008?