[05:33:46] I am going to stop replication on pc2009 and pc2010 and create a test database on it [06:07:13] 10DBA: Degraded performance on parsercache with buster/mariadb upgrade - https://phabricator.wikimedia.org/T252761 (10jcrespo) For memory-only read only traffic, regression seems to be only of 5%, which was around what we expected. Note pc1010 had a 13% average extra latency from client, so it is within the marg... [06:18:48] I am running a mixed read/write benchmark now [07:16:29] 10DBA: Degraded performance on parsercache with buster/mariadb upgrade - https://phabricator.wikimedia.org/T252761 (10jcrespo) These are my findings: `lines=5 average latency (ms) percentile 95 latency (ms) read requests per second write requests per second VERSION 10.1 10.4 10.1 10.4 10.1 10.4 10.1 10.4 ro,... [07:31:29] 10DBA: Degraded performance on parsercache with buster/mariadb upgrade - https://phabricator.wikimedia.org/T252761 (10jcrespo) These are the database metrics during the tests (tests were not concurrent between hosts): * pc1009: https://grafana.wikimedia.org/d/000000273/mysql?from=1589521050221&to=1589524843408&v... [08:50:58] 10DBA: Degraded performance on parsercache with buster/mariadb upgrade - https://phabricator.wikimedia.org/T252761 (10jcrespo) There is a difference in replication "performance" (pc1010 is spikier and lower): {F31819369} {F31819371} But the comparison is not fair for the 10.4 host, as it replicates from an inter... [09:47:17] 10DBA, 10Operations, 10Patch-For-Review: mysql user and group should be a system user/group - https://phabricator.wikimedia.org/T100501 (10jcrespo) 05Stalled→03Resolved a:03jcrespo All mysql users are system users. [09:47:21] 10DBA, 10Operations, 10Patch-For-Review: Prepare mysql hosts for stretch - https://phabricator.wikimedia.org/T168356 (10jcrespo) [10:20:14] 10DBA, 10Operations, 10Patch-For-Review: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10jcrespo) [10:20:58] 10DBA, 10Operations, 10Patch-For-Review: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10jcrespo) As per T162070#4942720. [10:43:41] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10jcrespo) 05Resolved→03Open @Papaul see the FAILED above for db2140, as well as the ` db2140 missing physical device in PuppetDB: state Staged in Net... [11:07:56] 10DBA, 10Operations, 10Patch-For-Review: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Dzahn) I have since gotten replies from 2 project owners and was able to remove the role from their instances. 2 are still missing b... [13:34:19] 10DBA, 10Patch-For-Review: Refactor transfer.py - https://phabricator.wikimedia.org/T252172 (10jcrespo) This can be resolved. [13:39:20] 10DBA, 10Patch-For-Review: Refactor transfer.py - https://phabricator.wikimedia.org/T252172 (10Privacybatm) 05Open→03Resolved Thank you for merging! :-) [13:42:55] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10Papaul) @jcrespo the server first boot was set to NIC1 and not Hard drive 1 so when it completed the OS install the first time re rebooted to PXE again it... [13:44:18] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` db2140.codfw.wmnet ` The log can be found in `/var... [13:44:22] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2140.codfw.wmnet'] ` Of which those **FAILED**: ` ['db2140.codfw.wmnet'] ` [13:44:31] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` db2140.codfw.wmnet ` The log can be found in `/var... [14:09:53] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2140.codfw.wmnet'] ` and were **ALL** successful. [14:10:39] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10jcrespo) 05Open→03Resolved Thanks, @Papaul [14:14:07] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 31st May) rack/setup/install db213[6-9] and db2140 - https://phabricator.wikimedia.org/T251639 (10Papaul) @jcrespo all good sorry about the problem [17:34:33] 10DBA, 10Operations, 10Patch-For-Review: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Dzahn) Removed simplelamp from the last 2 remaining projects listed in openstack-browser. I think it's good to go now. Commented on...