[06:38:18] 10DBA, 10Operations, 10ops-eqiad: db1114 crashed due to memory issues (server under warranty) - https://phabricator.wikimedia.org/T229452 (10Marostegui) 05Open→03Resolved The log is still clear. So closing this, if it happens again I will re-open ` /admin1-> racadm getsel Record: 1 Date/Time: 08/2... [08:48:49] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` dbproxy1019.eqiad.wmnet ` The log can be found in `/var/log/wmf-au... [09:07:19] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbproxy1019.eqiad.wmnet'] ` and were **ALL** successful. [09:20:13] 10DBA: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) dbproxy1019 is ready to take over dbproxy1011 ` root@dbproxy1019:~# echo "show stat" | socat /run/haproxy/haproxy.sock stdio # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq... [11:13:17] 10DBA, 10Operations, 10wikitech.wikimedia.org, 10Patch-For-Review, 10cloud-services-team (Kanban): Switchover m5 primary master: db1073 to db1133 - https://phabricator.wikimedia.org/T229657 (10Marostegui) @CDanis @Volans can you confirm this command will set wikitech (db1073 is its master) on read-only?:... [13:04:58] 10DBA, 10Operations, 10wikitech.wikimedia.org, 10Patch-For-Review, 10cloud-services-team (Kanban): Switchover m5 primary master: db1073 to db1133 - https://phabricator.wikimedia.org/T229657 (10Volans) >>! In T229657#5431230, @Marostegui wrote: > @CDanis @Volans can you confirm this command will set wikit... [13:39:01] 10DBA, 10Operations, 10wikitech.wikimedia.org, 10Patch-For-Review, 10cloud-services-team (Kanban): Switchover m5 primary master: db1073 to db1133 - https://phabricator.wikimedia.org/T229657 (10Marostegui) Indeed @Volans - thanks! ` root@cumin1001:~# dbctl --scope eqiad section wikitech rw && dbctl confi... [15:16:53] 10DBA, 10conftool: specify group (api/vslow/etc) weights in terms of 0..100 instead of 0..1 - https://phabricator.wikimedia.org/T231018 (10CDanis) [15:24:56] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10CDanis) [18:06:12] jynus: 10.1.41 is out with some interesting fixes: https://mariadb.com/kb/en/library/mariadb-10141-release-notes/ [19:32:55] meh, but they didn't fix the bug which breaks the PHP build, despite being tagged as a blocker https://phabricator.wikimedia.org/T230024#5428319 [19:44:06] moritzm: but it says fixed on that mariadb link [19:44:15] on 10.1.38 [19:57:21] that seems wrong, I ran into it with 10.1.41 on boron [19:57:54] in fact it got introduced with .38, it was still fine with .37 when i build PHP 7.2.16 some months ago [19:59:05] we run .39 in production, that one fails too? [20:47:22] we won't see that specific error, it's a bug mysql_install_db for our tools (but it's commonly used in Debian postinst scripts to create databases etc. [21:19:36] 10DBA: Okay to use `user_properties` table for high frequency/volume options? - https://phabricator.wikimedia.org/T231044 (10nray)