[05:34:46] 10DBA, 10Data-Services: Compress and defragment tables on labsdb hosts - https://phabricator.wikimedia.org/T222978 (10Marostegui) 05Open→03Resolved [05:51:30] 10DBA, 10Goal, 10Patch-For-Review: Productionize db21[21-30} - https://phabricator.wikimedia.org/T228969 (10Marostegui) [06:22:48] 10DBA, 10Operations, 10wikitech.wikimedia.org, 10cloud-services-team (Kanban): Switchover m5 primary master: db1073 to db1133 - https://phabricator.wikimedia.org/T229657 (10Marostegui) [06:31:39] 10DBA, 10Operations, 10wikitech.wikimedia.org, 10cloud-services-team (Kanban): Switchover m5 primary master: db1073 to db1133 - https://phabricator.wikimedia.org/T229657 (10Marostegui) Announcement about wikitech sent to wikitech-l and operations list: https://lists.wikimedia.org/pipermail/wikitech-l/2019-... [06:35:56] 10DBA, 10Goal: Productionize db21[21-30} - https://phabricator.wikimedia.org/T228969 (10Marostegui) [06:55:47] 10DBA, 10Operations, 10Patch-For-Review: Firewall configurations for database hosts - https://phabricator.wikimedia.org/T104699 (10MoritzMuehlenhoff) 05Open→03Resolved This is complete. [07:31:13] 10DBA, 10Operations, 10ops-eqiad: Upgrade db1100 firmware and BIOS - https://phabricator.wikimedia.org/T228732 (10Marostegui) @Cmjohnson db1100 is now OFF. Once you are done leave it ON and will take care of MySQL and the rest of things Thanks! [08:46:28] 10DBA, 10Goal, 10Patch-For-Review: Productionize db21[21-30} - https://phabricator.wikimedia.org/T228969 (10Marostegui) [09:49:03] 10DBA, 10Goal, 10Patch-For-Review: Productionize db21[21-31} - https://phabricator.wikimedia.org/T228969 (10Marostegui) [10:19:55] 10DBA, 10DC-Ops, 10Operations, 10decommission, 10ops-codfw: Decommission db2035 - https://phabricator.wikimedia.org/T229784 (10Marostegui) [11:01:25] 10DBA: Clean up "nobarrier" mount options for Buster - https://phabricator.wikimedia.org/T229915 (10Marostegui) [11:01:34] 10DBA: Clean up "nobarrier" mount options for Buster - https://phabricator.wikimedia.org/T229915 (10Marostegui) p:05Triage→03Normal [11:07:54] Amir1: I am seeing reads spikes again on db1109 [11:07:59] every two hours [11:08:10] https://grafana.wikimedia.org/d/000000273/mysql?panelId=34&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1109&var-port=9104&from=1565042462398&to=1565089616803 [11:08:24] https://grafana.wikimedia.org/d/000000273/mysql?panelId=3&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1109&var-port=9104&from=1565042462398&to=1565089616803 [11:08:33] :( this is external, I should let the user know to stop it [11:09:12] :( [11:11:39] let me dig hadoop [11:16:59] It's so funny that the time is actually changed, it's obviously being done manually at start of every week [11:17:10] and the person pulls the plug for the weekened [11:20:16] haha yeah [11:55:29] I couldn't get anything out of hadoop. In the timestamp, everything is normal, the usual offenders are on top, and they stay on top in other times [11:55:41] my only suspicion is that I'm reading the times wrong [11:56:49] 10DBA: Clean up "nobarrier" mount options for Buster - https://phabricator.wikimedia.org/T229915 (10MoritzMuehlenhoff) The nobarrier option wasn't ever supported in the Debian installer. partman-xfs only supports the following options and the last change to that file was 12 years ago :-) https://salsa.debian.org... [11:57:43] 10DBA: Clean up "nobarrier" mount options for Buster - https://phabricator.wikimedia.org/T229915 (10Marostegui) Thanks :) [11:58:04] nope, my times are correct [12:20:53] 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10Marostegui) [13:59:59] 10DBA, 10Patch-For-Review: Clean up "nobarrier" mount options for Buster - https://phabricator.wikimedia.org/T229915 (10Marostegui) 05Open→03Resolved a:03MoritzMuehlenhoff Thanks for the patch and the merge! [14:31:36] 10DBA, 10Operations, 10ops-eqiad: Upgrade db1100 firmware and BIOS - https://phabricator.wikimedia.org/T228732 (10Cmjohnson) 05Open→03Resolved @marostegui The f/w and bios update is complete. [14:31:39] 10DBA, 10Operations: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Cmjohnson) [14:36:20] 10DBA, 10Operations, 10ops-eqiad: Upgrade db1100 firmware and BIOS - https://phabricator.wikimedia.org/T228732 (10Marostegui) Thanks - I will start MySQL now and take it from there [14:38:30] 10DBA, 10Operations: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui) [15:00:31] 10DBA, 10Operations, 10ops-codfw: (2019-08-31)rack/setup/install db2131.codfw.wmnet - https://phabricator.wikimedia.org/T229251 (10Papaul) a:05Papaul→03Marostegui Fixed [15:37:13] marostegui: it's not happening anymore [15:38:10] you talked to the user? [15:38:16] or they had a break? :) [15:57:43] They are just having a break I guess [15:57:55] I couldn't find anything out of ordinary in hadoop [15:58:09] I did it for another time span and still the same [15:58:21] I'm honestly lost on what to do [15:58:27] https://grafana.wikimedia.org/d/000000548/wikibase-wb_terms?refresh=30s&orgId=1&from=now-7d&to=now-5d [15:58:36] It's definitely not as bad as last week [16:09:20] marostegui: For now, we add a layer of cache on property terms, that would reduce a big amount of reads [16:11:06] sounds good :) [16:11:08] thanks [18:20:37] 10DBA, 10AbuseFilter, 10Patch-For-Review, 10Schema-change: Move AbuseFilter slow filters data from Logstash to per-filter profiling - https://phabricator.wikimedia.org/T179604 (10Krinkle)