[09:17:11] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320#4199524 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db1066.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimage/201... [11:34:01] 10DBA, 10Data-Services, 10Patch-For-Review: Issues with load on wikireplicas - https://phabricator.wikimedia.org/T194343#4200137 (10jcrespo) The changes I made (I also optimized some tables) multiplied by 2x-3x the number of writes the hosts can do, but we do not know yet what is the root cause. The issues d... [13:05:57] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320#4200236 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db1076.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimage/201... [13:26:24] 10DBA, 10Patch-For-Review: Decommission db1051-db1060 (DBA tracking) - https://phabricator.wikimedia.org/T186320#4200284 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1076.eqiad.wmnet'] ``` and were **ALL** successful. [15:10:03] 10DBA, 10Data-Services, 10Patch-For-Review: Issues with load on wikireplicas - https://phabricator.wikimedia.org/T194343#4200496 (10jcrespo) I found the issue, user s51559 was running multiple concurrent runs of (the query didn't fit in my buffer): ```lines=10 ev_minor_edit`,`page_title`,`page_namespace`,`p... [15:16:40] 10DBA, 10Data-Services, 10Toolforge, 10Tracking: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#4200522 (10jcrespo) [15:40:29] 10DBA, 10Data-Services, 10Toolforge, 10Tracking: Certain tools users create multiple long running queries that take all memory and/or CPU from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#4200560 (10jcrespo) [16:07:31] 10DBA, 10Data-Services, 10Patch-For-Review: Issues with load on wikireplicas - https://phabricator.wikimedia.org/T194343#4200608 (10bd808) >>! In T194343#4200496, @jcrespo wrote: > I found the issue, user s51559 was running multiple concurrent runs of (the query didn't fit in my buffer): That giant query lo... [16:08:26] 10DBA, 10Data-Services, 10Patch-For-Review: quentinv57-tools/tools/globalcontribs.php generates slow/complex SQL queries which impact server performance - https://phabricator.wikimedia.org/T194343#4200611 (10bd808) [16:12:15] 10DBA, 10Data-Services, 10Tool-Quentinv57's-tools, 10Patch-For-Review: quentinv57-tools/tools/globalcontribs.php generates slow/complex SQL queries which impact server performance - https://phabricator.wikimedia.org/T194343#4200628 (10JJMC89) [16:45:07] 10DBA, 10Packaging: Bug on mariadb systemd unit on stretch for multi-instance hosts - https://phabricator.wikimedia.org/T194516#4200657 (10jcrespo) [16:45:17] 10DBA, 10Packaging: Bug on mariadb systemd unit on stretch for multi-instance hosts - https://phabricator.wikimedia.org/T194516#4200669 (10jcrespo) p:05Triage>03Normal [16:46:47] 10DBA, 10Packaging: Bug on mariadb systemd unit on stretch for multi-instance hosts - https://phabricator.wikimedia.org/T194516#4200657 (10jcrespo) Patching the unit manually doesn't work, because the state of the config at start is remembered, requiring a rolling restart. [16:59:52] 10DBA, 10Packaging: Bug on mariadb systemd unit on stretch for multi-instance hosts - https://phabricator.wikimedia.org/T194516#4200715 (10jcrespo) This will require a 10.0.33-2, a systemctl daemon-reload and then a restart of all services. [17:13:36] 10DBA, 10Operations, 10ops-eqiad: Move db1066 to row A - https://phabricator.wikimedia.org/T193847#4181453 (10jcrespo) db1066 is now pooled on s2, so it will need a depooling before shutting it down (and probably disk changes). [17:27:29] 10DBA, 10Epic: Meta ticket: Migrate multi-source database hosts to multi-instance - https://phabricator.wikimedia.org/T159423#4200851 (10jcrespo) [17:27:32] 10DBA, 10Packaging: Bug on mariadb systemd unit on stretch for multi-instance hosts - https://phabricator.wikimedia.org/T194516#4200850 (10jcrespo) [23:45:32] 10DBA, 10Data-Services, 10Tool-Quentinv57's-tools, 10Patch-For-Review: quentinv57-tools/tools/globalcontribs.php generates slow/complex SQL queries which impact server performance - https://phabricator.wikimedia.org/T194343#4201901 (10MarcoAurelio) @jcrespo I am not responsible in any way on this issue. I...