[00:09:25] 10DBA, 10Operations, 10Performance-Team, 10Availability (Multiple-active-datacenters): Perform testing for TLS effect on connection rate - https://phabricator.wikimedia.org/T171071#3888608 (10aaron) I fixed a stupid hostname var bug. Now I get numbers that make sense: ``` Same-DC (db2070.codfw.wmnet): stri... [00:48:52] 10DBA, 10Data-Services, 10User-bd808, 10cloud-services-team (Kanban): Determine schema differences between labsdb1001 and labsdb1009 - https://phabricator.wikimedia.org/T177223#3888671 (10bd808) Looking over the diffs again, I think these are the potential indices to add: ```lang=sql -- exists in Educatio... [06:25:38] 10DBA, 10Data-Services, 10User-bd808, 10cloud-services-team (Kanban): Determine schema differences between labsdb1001 and labsdb1009 - https://phabricator.wikimedia.org/T177223#3888956 (10Marostegui) The index `ep_articles_course_id` was deleted per: T180166 The index wb_ips_site_page was deleted per: T179... [06:29:36] 10DBA, 10Data-Services, 10Patch-For-Review: Some queries to new replica hosts are dramatically slower than labsdb; missing indexes? - https://phabricator.wikimedia.org/T177096#3888961 (10bd808) [06:29:39] 10DBA, 10Data-Services, 10User-bd808, 10cloud-services-team (Kanban): Determine schema differences between labsdb1001 and labsdb1009 - https://phabricator.wikimedia.org/T177223#3888959 (10bd808) 05Open>03Resolved Per T177223#3888671 and T177223#3888956 it seems like we are in parity. [07:38:00] 10DBA: Drop `external_user` from all databases - https://phabricator.wikimedia.org/T184247#3888996 (10Marostegui) ``` root@db1071[wikidatawiki]> select count(*) from external_user; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.00 sec) root@db1071[wikidatawiki]> drop table if e... [07:38:34] 10DBA: Drop `external_user` from all databases - https://phabricator.wikimedia.org/T184247#3889000 (10Marostegui) [08:10:04] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3889042 (10Marostegui) s5 codfw is done [08:10:14] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3889043 (10Marostegui) [08:13:19] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3889045 (10Marostegui) Tracking s5 eqiad here: [] labsdb1001 (broken, read-only, will not get it) [] labsdb1003 (stopped, will not... [08:16:36] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3889048 (10Marostegui) [10:08:17] I'm installing PHP security updates on dbmonitor1001, there might be a short hiccup for tendril.w.o [10:08:50] thanks for the heads up [10:09:30] done [10:17:49] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018): Determine how to update old compressed ExternalStore entries for T181555 - https://phabricator.wikimedia.org/T183419#3889335 (10jcrespo) That looks ok to me -config has to be added to codfw in parallel, but otherwise it is ok-, please keep me updated whe... [10:30:30] 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#3889371 (10jcrespo) > Is mysql-upgrade going to ensure it doesn't run while anything else is doing DDL Yes, because we are the only... [10:40:54] 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#3889392 (10jcrespo) > Could you help us figure out how we can better detect this so we can script it in? Thank you! I don't think th... [11:11:53] 10DBA, 10Operations, 10Performance-Team, 10Availability (Multiple-active-datacenters): Perform testing for TLS effect on connection rate - https://phabricator.wikimedia.org/T171071#3889432 (10jcrespo) So actually, that is not really that bad- query times are similar (only some small overhead), connection t... [11:15:35] 10DBA, 10Operations, 10Performance-Team, 10Availability (Multiple-active-datacenters): Perform testing for TLS effect on connection rate - https://phabricator.wikimedia.org/T171071#3889438 (10jcrespo) One thing I just realized is that there could be some connection overhead on db1055- I will (or you can) t... [11:29:46] 10DBA: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#3889458 (10jcrespo) [12:05:34] I wonder if we should move labswiki to s5 rather than s3? [12:05:57] I am fine with that [12:06:01] we should discuss with releng [12:06:33] as maybe there is some expectation than group0 wikis that can break are all on the same replica group? [12:06:34] I don't think they would care much about where we place it, no? [12:06:36] Aaaah [12:06:47] Maybe they do care then :) [12:06:49] I know mediawikiwiki and others [12:06:53] more like, we care [12:07:14] if s5 can break, we may not want it with other high profile wikis [14:23:04] I am goign to start cleaning up s5/s8 tables on dbstore2001 [14:28:00] it is strange, there is a jump from 20180109060123 to 20180109061517 [14:28:28] but then there is some 2000 estra rows, up to 20180109081633 [14:28:32] *extra [14:30:30] oh, actually it is ok [14:30:41] it is a leftover of doing the copy later [14:32:11] it is not happening on production [14:32:14] so expected [14:43:21] dropping wikidata on HDs is not precisely fast [15:31:42] jynus: meeting? [15:32:03] which one? [15:32:21] I just saw that [15:32:26] I got no email [15:47:28] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3890015 (10Cmjohnson) [15:47:31] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1044 - https://phabricator.wikimedia.org/T181696#3890014 (10Cmjohnson) 05Open>03Resolved [15:47:48] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2713121 (10Cmjohnson) [15:47:51] 10DBA, 10Operations, 10Phabricator, 10hardware-requests, 10ops-eqiad: Decommission db1048 (was Move m3 slave to db1059) - https://phabricator.wikimedia.org/T175679#3890016 (10Cmjohnson) 05Open>03Resolved [15:48:05] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2714212 (10Cmjohnson) [15:48:08] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1015 - https://phabricator.wikimedia.org/T173570#3890019 (10Cmjohnson) 05Open>03Resolved [15:48:20] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1049 - https://phabricator.wikimedia.org/T175264#3890021 (10Cmjohnson) 05Open>03Resolved [15:48:23] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2714241 (10Cmjohnson) [15:48:37] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1045 - https://phabricator.wikimedia.org/T174806#3890023 (10Cmjohnson) 05Open>03Resolved [15:48:52] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1026 - https://phabricator.wikimedia.org/T174763#3890024 (10Cmjohnson) 05Open>03Resolved [15:49:08] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#2767459 (10Cmjohnson) [15:49:11] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1021 - https://phabricator.wikimedia.org/T181378#3890025 (10Cmjohnson) 05Open>03Resolved [15:50:29] 10DBA, 10Operations, 10ops-eqiad: db1059 possibly BBU issues - https://phabricator.wikimedia.org/T184160#3890028 (10Cmjohnson) @marostegui II have a used spare battery we can swap this out with. LMK when you want to schedule this [15:52:02] 10DBA, 10Operations, 10ops-eqiad: db1059 possibly BBU issues - https://phabricator.wikimedia.org/T184160#3890043 (10Marostegui) @Cmjohnson you want me to power off the server and we can do it now? [15:54:48] 10DBA, 10Operations, 10ops-eqiad: db1059 possibly BBU issues - https://phabricator.wikimedia.org/T184160#3890063 (10Marostegui) As per our chat, this will be done tomorrow [15:56:47] 10DBA, 10Multi-Content-Revisions, 10Epic, 10Schema-change: Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields to be dropped from revision - https://phabricator.wikimedia.org/T184615#3890073 (10Jdforrester-WMF) 05Open>03stalled p:05Triage>03Low [15:56:59] 10DBA, 10Multi-Content-Revisions, 10Epic, 10Schema-change: Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields to be dropped from revision - https://phabricator.wikimedia.org/T184615#3890084 (10Jdforrester-WMF) [15:57:06] 10DBA, 10Multi-Content-Revisions, 10Epic, 10Schema-change: Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields to be dropped from revision - https://phabricator.wikimedia.org/T184615#3890073 (10Jdforrester-WMF) [16:00:47] I honestly didn't receive an invitation [16:04:51] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3890127 (10Marostegui) >>! In T174569#3890126, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), hre... [16:27:02] 10DBA: s5 wikidatawiki database cleanup - https://phabricator.wikimedia.org/T184599#3890164 (10jcrespo) [16:58:20] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2060 - https://phabricator.wikimedia.org/T184464#3890327 (10Papaul) a:05Papaul>03Marostegui Disk replacement complete [17:00:30] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2060 - https://phabricator.wikimedia.org/T184464#3890357 (10Marostegui) Thanks - will close once it has finished: ``` logicaldrive 1 (3.3 TB, RAID 1+0, Recovering, 3% complete) physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, Rebuilding)... [17:24:51] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018): Determine how to update old compressed ExternalStore entries for T181555 - https://phabricator.wikimedia.org/T183419#3852480 (10Anomie) 05Open>03Resolved Ok, since we've determined how to do it I'm going to resolve this task. I copied the plan into t... [18:19:25] 10DBA, 10Operations, 10Patch-For-Review: Setup newer machines and replace all old misc (m*) and x1 eqiad machines - https://phabricator.wikimedia.org/T183469#3890726 (10jcrespo) a:03jcrespo [18:20:16] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3890733 (10jcrespo) [18:20:20] 10DBA, 10Operations, 10Patch-For-Review: Firewall configurations for database hosts - https://phabricator.wikimedia.org/T104699#3890734 (10jcrespo) [18:20:23] 10DBA, 10Operations, 10Patch-For-Review: Reimage and upgrade to stretch all dbproxies - https://phabricator.wikimedia.org/T183249#3890730 (10jcrespo) 05Open>03stalled a:05jcrespo>03None [18:38:08] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2060 - https://phabricator.wikimedia.org/T184464#3890799 (10jcrespo) 05Open>03Resolved a:05Marostegui>03Papaul [19:14:02] 10DBA, 10Operations, 10ops-eqiad: db1059 BBU issues - https://phabricator.wikimedia.org/T184160#3890941 (10jcrespo) [19:38:52] 10DBA, 10Multi-Content-Revisions, 10Epic, 10Schema-change: Once MCR is deployed, drop the rev_text_id, rev_content_model, and rev_content_format fields to be dropped from revision - https://phabricator.wikimedia.org/T184615#3891004 (10kaldari) [20:11:10] 10DBA, 10Operations, 10Performance-Team, 10Availability (Multiple-active-datacenters): Perform testing for TLS effect on connection rate - https://phabricator.wikimedia.org/T171071#3453087 (10Imarlier) Is the goal here just to quantify the impact? Or is there a target connect time/query time that we're tr... [23:03:15] 10DBA, 10Data-Services, 10cloud-services-team: Maintain-views and maintain_meta-p scripts shouldn't run if mysql-upgrade is running - https://phabricator.wikimedia.org/T184540#3891670 (10bd808) >>! In T184540#3889392, @jcrespo wrote: >> Could you help us figure out how we can better detect this so we can scr...