[00:07:07] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4255644 (10dmaza) > What is the rough timeline to production? Any hard deadlines you know as of now (if you have th... [05:44:49] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4255833 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2... [05:57:12] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4255850 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db2... [06:26:59] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4255878 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db2059.codfw.wmnet'] ``` and were **ALL** successful. [07:23:32] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4255914 (10jcrespo) > This is one of our quarterly goals. We want to get on this ASAP and DB changes are the first... [07:23:53] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4255915 (10jcrespo) >> Also, are you aware of the actor tables reformatting, which will change how tables store ref... [07:45:49] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 2 others: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273#4255967 (10Lydia_Pintscher) [08:43:25] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on labsdb1009 - https://phabricator.wikimedia.org/T195690#4256161 (10MoritzMuehlenhoff) >>! In T195690#4254655, @Marostegui wrote: > I just saw it is in the repo but it wasn't installed > We should add it by default on the puppet recipe for HP hosts probably Y... [09:17:51] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376#4254041 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts: ``` ['db1120.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-re... [09:35:03] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376#4256430 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1120.eqiad.wmnet'] ``` and were **ALL** successful. [09:35:29] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy - https://phabricator.wikimedia.org/T190704#4256433 (10Marostegui) [09:35:33] 10DBA, 10Patch-For-Review: Productionize old/temporary eqiad sanitariums - https://phabricator.wikimedia.org/T196376#4256431 (10Marostegui) 05stalled>03Open [09:54:19] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4256471 (10Marostegui) [10:25:19] https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180613T0600 [10:27:42] Oh, thanks! [11:00:41] random question of the day [11:01:05] is there a plan around dbstore1002 for its deprecation or replacement or something? [11:01:10] yes [11:01:23] however you are asking the wrong team [11:01:32] that is owned by analytics [11:02:29] if it was up to me, I would have shut down that a long time ago [11:03:33] ok [11:03:35] is there a task? [11:03:39] do you happen to know? [11:04:07] paravoid: this is the last thing I know: https://phabricator.wikimedia.org/T159423#3977758 [11:04:54] alright [11:04:56] thank you [11:07:43] paravoid: if you saw our report of stretch reimage, we alway separate dbstore1002 because things are fuzzy there- maybe it needs a stewardship process [11:08:25] (not about the database, we don't have any problem maintaining that, but about the service it provides) [11:09:08] we cannot ever do proper maintenance because of that [13:34:56] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4257526 (10dbarratt) >>! In T193449#4255644, @dmaza wrote: > If any of the previous queries return a valid block fo... [15:01:47] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4257779 (10Anomie) >>! In T193449#4255915, @jcrespo wrote: >>> Also, are you aware of the actor tables reformatting... [15:07:19] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4257801 (10jcrespo) ^Basically, +1 what anomie says > I've heard a proposal for a "titles" table That could be me... [15:17:15] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: pc2005 down - https://phabricator.wikimedia.org/T196339#4257846 (10Papaul) Will be receiving a main board replacement . Enterprise Service Request 966671345 [15:17:40] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: pc2005 down - https://phabricator.wikimedia.org/T196339#4257847 (10Papaul) Good morning Papaul, I set up a dispatch for a replacement motherboard and daughter card set for parts only as requested. I would recommend replacing the daughter card first... [15:18:58] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: pc2005 down - https://phabricator.wikimedia.org/T196339#4257848 (10jcrespo) yay! for the replacements, sorry to create you more work. [15:21:01] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T196246#4257853 (10Papaul) @Marostegui since db2067 is out of service, would you like for me to take one disk from it and replace the failed disk on db2047? [15:23:19] ^I would say yes, but your call (because of test) [15:23:23] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: pc2005 down - https://phabricator.wikimedia.org/T196339#4257855 (10Papaul) @jcrespo you didn't create me more work, it is my work lol [15:23:29] I deployed the extra port patch [15:23:38] was a bit bumpy, but it went through [15:23:55] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T196246#4257856 (10Marostegui) @Papaul db2067 is online, it is not out of service. You sure you meant db2067? [15:25:30] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T196246#4251009 (10jcrespo) He meant (probably) db2064 [15:26:30] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T196246#4257862 (10Papaul) db2064 thanks @jcrespo [15:28:27] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T196246#4257863 (10Marostegui) Sure then, if the disks are the same (600GB) :-) [15:31:41] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T196246#4257873 (10Papaul) a:05Papaul>03Marostegui Disk replacement complete. [15:32:49] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T196246#4257877 (10Marostegui) Thanks! ``` physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, Rebuilding) ``` [15:41:26] how are things going? [15:41:30] with labsdb1005? [15:41:45] it is all done [15:42:04] good, thanks [15:42:17] :) [16:44:02] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4258153 (10dbarratt) >>! In T193449#4257779, @Anomie wrote: > With either option, I note there's no allowance for b... [17:09:44] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4258251 (10Anomie) As long as it's planned for rather than something overlooked, I don't have an issue with it. Mo... [17:14:30] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on labsdb1009 - https://phabricator.wikimedia.org/T195690#4258274 (10Cmjohnson) The report was sent to HP yesterday, i have not heard back from them yet. If I don't get something in the next few hours I will ping them. [17:17:03] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on labsdb1009 - https://phabricator.wikimedia.org/T195690#4258276 (10Marostegui) Thanks for the update! [17:53:06] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1065 - https://phabricator.wikimedia.org/T196490#4258498 (10Marostegui) This disk has been replaced by @Cmjohnson as we were coming from: T195444#4230827 [17:53:37] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1065 - https://phabricator.wikimedia.org/T196490#4258505 (10Marostegui) p:05Triage>03Normal [17:55:44] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1065 - https://phabricator.wikimedia.org/T196490#4258525 (10Marostegui) ``` root@db1065:~# megacli -PDRbld -ShowProg -PhysDrv [32:1] -aALL Rebuild Progress on Device at Enclosure 32, Slot 1 Completed 56% in 31 Minutes. ``` [18:06:19] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2047 - https://phabricator.wikimedia.org/T196246#4258595 (10Marostegui) 05Open>03Resolved All good! ``` logicaldrive 1 (3.3 TB, RAID 1+0, OK) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK) physicaldrive 1I:1:2 (port 1I:b... [18:58:55] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on labsdb1009 - https://phabricator.wikimedia.org/T195690#4258824 (10Cmjohnson) This is Regarding the Case Number 5329764075 for HPE ProLiant DL380 Gen9 8SFF Configure-to-order Server Issue: SCM_HW:Failed Hard Drive Thanks a lot for sharing the Smart Wear G... [19:03:51] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4258836 (10dbarratt) Well, as I mentioned in T193449#4237241, I prefer //Option B//. It seems like the greatest va... [19:07:37] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4258855 (10dbarratt) Also, as mentioned in T193449#4257526, I think we should add a boolean (tinyint) field to `ipb... [21:25:55] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4259305 (10kaldari) @dbarratt: What, in your view, is the downside of using Option A? [22:23:37] 10DBA, 10MediaWiki-User-management, 10Anti-Harassment (AHT Sprint 21/22): Draft a proposal for granular blocks table schema(s), submit for DBA review - https://phabricator.wikimedia.org/T193449#4259415 (10dbarratt) >>! In T193449#4259305, @kaldari wrote: > @dbarratt: What, in your view, is the downside of us...