[06:36:09] 10DBA: Drop wikidatawiki.wb_items_per_site_old from s8 hosts - https://phabricator.wikimedia.org/T250345 (10Marostegui) [06:36:56] 10DBA: Drop wikidatawiki.wb_items_per_site_old from s8 hosts - https://phabricator.wikimedia.org/T250345 (10Marostegui) Removal progress: [] codfw (to be done with replication) [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1005 [] db1126 [] db1124 [] db1116 [] db1114 [] db1111 [] db1109 [] d... [06:39:38] 10DBA, 10MediaWiki-Page-derived-data, 10Schema-change: Avoid MySQL's ENUM type, which makes keyset pagination difficult - https://phabricator.wikimedia.org/T119173 (10jcrespo) I think it is wrong to blankly ban enum usage. Sure, ORDER BY uses its internal id, but that is a widely known feature of MySQL enums... [06:41:14] hahahah I was essentially writing the same thing [06:41:16] ^ [06:41:48] feel free to still do it- I just wrote my opinion [07:25:27] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Implement logic to be able to perform full and incremental backups of ES hosts - https://phabricator.wikimedia.org/T244884 (10jcrespo) 05Open→03Stalled [07:25:32] 10DBA, 10Operations, 10Goal: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10jcrespo) [07:25:49] 10DBA, 10Operations, 10Patch-For-Review: Implement logic to be able to perform full and incremental backups of ES hosts - https://phabricator.wikimedia.org/T244884 (10jcrespo) [07:40:31] 10DBA: Drop wikidatawiki.wb_items_per_site_old from s8 hosts - https://phabricator.wikimedia.org/T250345 (10Kormat) a:03Kormat [07:40:35] woooooot [07:40:45] damn, now there's a public papertrail [07:41:49] 10DBA, 10Operations: replace phabricator db passwords with longer passwords - https://phabricator.wikimedia.org/T250361 (10fgiunchedi) p:05Triage→03Medium [07:43:14] 10DBA: Drop wikidatawiki.wb_items_per_site_old from s8 hosts - https://phabricator.wikimedia.org/T250345 (10jcrespo) [07:46:13] <_joe_> kormat: there's always a paper trail here [07:46:19] <_joe_> including this channel [09:27:47] 10DBA: Drop wikidatawiki.wb_items_per_site_old from s8 hosts - https://phabricator.wikimedia.org/T250345 (10Kormat) 05Open→03Resolved It's gone: ` root@cumin1001:~# ~kormat/section s8 | while read host port; do echo "=====> $host"; mysql.py -h $host -P $port wikidatawiki -e 'show tables like "wb_items_per_si... [09:28:41] 10DBA: Drop wikidatawiki.wb_items_per_site_old from s8 hosts - https://phabricator.wikimedia.org/T250345 (10Marostegui) \o\ |o| /o/ [09:30:23] with enough luck, he will hate the process enough to automate it! [09:30:42] he actually mentioned it already [09:30:43] haha [09:31:15] https://i.pinimg.com/originals/4e/6c/14/4e6c14cb945754074a58efef007d9bad.jpg [09:31:38] * kormat sobs quietly [10:13:36] 10DBA, 10Operations, 10Wikimedia-Incident: investigate pc1008 for possible hardware issues / performance under high load - https://phabricator.wikimedia.org/T247787 (10Kormat) Magic happened: - Optimized tables on pc1010 to free up some space (it halved disk usage) - Moved pc1010 to replicate under pc1007 -... [11:08:59] 10DBA, 10Operations, 10Goal: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10jcrespo) Proposed backup workflow strategy for content and metadata database backups: {F31761425} [11:09:18] ^let me know what you think [11:11:39] 10DBA, 10Operations, 10Goal: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10Marostegui) Does that mean we won't be storing metadata/content backup within the same DC and only on the remote DC? [11:11:41] just commented :) [11:11:46] Going to work out a bit [11:11:50] let's chat later on the task! [11:28:48] 10DBA, 10Operations, 10Goal: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10jcrespo) > Does that mean we won't be storing metadata/content backup within the same DC and only on the remote DC There will be backups available within the same dc, the freshest ones, on dbpro... [11:39:57] 10DBA, 10Operations, 10Goal: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10jcrespo) The main question is not really if we should do this or not- we should have bacula redundancy among dcs, the question is if we should cross-dc send the bacula copies or be 100% independe... [12:20:57] 10DBA, 10Operations, 10Goal: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10Marostegui) Thanks for the clarification, from the diagram it wasn't entirely clear to me if the short-term backups would be kept locally (hence my question :-) ) or not. I also like more the i... [13:19:37] 10DBA, 10Operations, 10Goal: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10jcrespo) > the diagram it wasn't entirely clear The summary is: "generate and keep short term locally, store long term on a geographically separate site." [13:39:31] 10DBA, 10Operations, 10Goal: Set up backup strategy for es clusters - https://phabricator.wikimedia.org/T79922 (10Marostegui) Yep, that is good - thanks for the clarification :-) [20:04:48] 10DBA, 10MediaWiki-Page-derived-data, 10Performance-Team (Radar), 10Schema-change: Avoid MySQL's ENUM type, which makes keyset pagination difficult - https://phabricator.wikimedia.org/T119173 (10Krinkle) a:03Krinkle [20:15:13] 10DBA, 10Core Platform Team Workboards (Clinic Duty Team), 10Performance Issue: Consider enabling max_statement_time on all production DB servers to limit impact of bad queries - https://phabricator.wikimedia.org/T250527 (10Jdforrester-WMF) p:05Triage→03High [20:16:02] 10DBA, 10Core Platform Team Workboards (Clinic Duty Team), 10Performance Issue: Consider enabling max_statement_time on all production DB servers to limit impact of bad queries - https://phabricator.wikimedia.org/T250527 (10Jdforrester-WMF) (Ahead of the idea in T195792 of managing this from MediaWiki.)