[08:54:30] <jynus>	 manuel, I would like your explicit ok on https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/442250 mariadb
[08:54:46] <marostegui>	 checkingh
[08:55:01] <jynus>	 not on the change, but on what that implies
[08:55:06] <jynus>	 after deployed
[08:55:08] <marostegui>	 Ah, I was actually going to update the ticket about the failover, saying that we should go for it
[08:55:15] <marostegui>	 I thought about it yesterday
[08:55:22] <marostegui>	 that we have had s2 master running with no issues
[08:55:44] <jynus>	 ok, doing it then
[08:55:53] <marostegui>	 +1!
[08:55:55] <jynus>	 heads up for reduced redundancy while it happens (today)
[08:56:29] <marostegui>	 yeah, and we need to think about another candidate master for s1
[08:56:35] <marostegui>	 but we can do that once the failover has happened
[08:56:49] <jynus>	 looking we will remove db1067 on Q3
[08:57:27] <jynus>	 probably Q4, actually
[08:57:33] <jynus>	 that is june 2019
[08:58:08] <jynus>	 BTW, I just uploaded 10.1.34
[08:58:35] <marostegui>	 \o/
[10:16:21] <wikibugs>	 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069#4278128 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts: ``` ['db1067.eqiad.wmnet'] ``` The log can be found in `/var/log/wmf-auto-reimage/201806...
[10:37:47] <wikibugs>	 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069#4318716 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['db1067.eqiad.wmnet'] ```  and were **ALL** successful.
[10:38:11] <marostegui>	 \o/
[10:48:22] <jynus>	 I am getting this error
[10:48:27] <jynus>	 [ERROR] Can't open shared library '/opt/wmf-mariadb101/lib/plugin/ha_sample.so' (errno: 0, cannot open shared object file: No such file or directory)
[10:48:40] <marostegui>	 on the new .1.34?
[10:51:47] <jynus>	 yes, although maybe it happened before
[10:52:36] <marostegui>	 I am reading the changelog for that release
[10:52:41] <marostegui>	 To see if there is anything related
[10:57:18] <jynus>	 I want to know exactly what that library is
[10:58:13] <jynus>	 I think it is the SAMPLE storage engine https://dev.mysql.com/doc/refman/8.0/en/example-storage-engine.html
[10:58:40] <jynus>	 which we could ignore
[10:58:49] <jynus>	 but 1) why is it being loaded
[10:58:53] <marostegui>	 yeah: https://mariadb.com/kb/en/library/show-plugins-soname/
[10:58:58] <jynus>	 2) why is it missing from my package
[10:59:20] <jynus>	 as basically, I include all plugins
[11:00:06] <jynus>	 there is a ha_example.so
[11:00:09] <jynus>	 though
[11:00:14] <marostegui>	 i was going to ask that
[11:00:22] <marostegui>	 if it was generated during the compilation
[11:00:28] <jynus>	 big if
[11:00:40] <jynus>	 I will compare to a lower version
[11:00:43] <jynus>	 and to 8.0
[11:01:24] <jynus>	 ha_sample wasn't there on 10.0
[11:01:30] <marostegui>	 and on 10.1.33?
[11:01:40] <jynus>	 do you have an example?
[11:02:31] <marostegui>	 db1076
[11:02:36] <jynus>	 it wasn't there
[11:02:43] <jynus>	 maybe the error wasn't new
[11:02:51] <marostegui>	 it is not there, no
[11:03:26] <jynus>	 but I see no error
[11:03:35] <jynus>	 maybe it was a db1067 only error
[11:04:02] <jynus>	 not sure what I should do
[11:04:10] <marostegui>	 It is very weird
[11:04:23] <jynus>	 I don't think it is going to break anything
[11:04:32] <marostegui>	 No, probably not
[11:04:47] <marostegui>	 I guess we should include it and that's it
[11:04:47] <jynus>	 but I don't like it
[11:04:59] <jynus>	 I think it was renamed
[11:05:03] <jynus>	 at some point
[11:05:10] <marostegui>	 oh wait
[11:05:12] <marostegui>	 db1076 does have it
[11:05:15] <marostegui>	 ha_example.so
[11:05:22] <jynus>	 yes
[11:05:26] <jynus>	 but not sample
[11:05:31] <marostegui>	 yeah, that is what I mean
[11:05:35] <jynus>	 ha_sample.so
[11:05:35] <marostegui>	 that it is renamed as you said
[11:09:11] <jynus>	 but it errored again after restart
[11:09:20] <jynus>	 so could it be configured to load it?
[11:09:34] <marostegui>	 what do you mean?
[11:09:39] <jynus>	 I can also run uninstall plugin
[11:09:57] <jynus>	 oh
[11:10:03] <jynus>	 I figured out
[11:10:18] <marostegui>	 what is it then!
[11:10:34] <jynus>	 https://phabricator.wikimedia.org/P7305
[11:10:39] <jynus>	 it was loaded in the past
[11:10:45] <jynus>	 so it tries to load it again
[11:10:53] <marostegui>	 aaaah right!
[11:11:47] <jynus>	 I am not sure I can uninstall a plugin it is not installed
[11:11:53] <jynus>	 so I will just delete the row
[11:12:13] <marostegui>	 yeah
[11:12:26] <marostegui>	 so in a fresh new server it should just work fine
[11:12:35] <marostegui>	 but if we are migrating from 10.0 it will always fail then
[11:12:46] <jynus>	 I think it is only db1067
[11:13:11] <jynus>	 which at some poing in the past someone loaded it
[11:13:18] <marostegui>	 I was checking db1068 and it is not loaded
[11:13:18] <jynus>	 maybe on 5.5 version or something
[11:13:26] <marostegui>	 (which is 10.0)
[11:13:37] <marostegui>	 neither db1075 (10.0)
[11:14:26] <jynus>	 I think that should be it
[11:15:04] <jynus>	 there is also [ERROR] Plugin 'rpl_semi_sync_slave' already installed
[11:15:24] <jynus>	 but that is our configuration, which cannot yet use the INSTALL IGNORE equivalent
[11:16:01] <marostegui>	 yeah, that should be fine
[11:21:52] <wikibugs>	 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069#4318841 (10jcrespo)
[11:24:23] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4318851 (10Marostegui)
[13:32:54] <wikibugs>	 10DBA, 10Data-Services, 10Tool-Quentinv57's-tools, 10Patch-For-Review: quentinv57-tools/tools/globalcontribs.php generates slow/complex SQL queries which impact server performance - https://phabricator.wikimedia.org/T194343#4319304 (10Pipetricker)
[16:39:04] <wikibugs>	 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4319776 (10Marostegui)
[16:39:07] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4319778 (10Marostegui)
[16:39:09] <wikibugs>	 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926#4319777 (10Marostegui)
[16:39:13] <wikibugs>	 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review, 10Wikidata-Ministry-Of-Magic: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193#4319779 (10Marostegui)
[20:18:17] <wikibugs>	 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#4320431 (10Ladsgroup) Seco...
[21:07:01] <wikibugs>	 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4320537 (10dduvall) Pinging #dba on this since the symptoms include lock timeouts.
[22:01:56] <wikibugs>	 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4320155 (10daniel) Is that stack trace representative? is it always INSERT  INTO `revision_comment_temp`?   It's quite possib...
[22:34:42] <wikibugs>	 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4320661 (10Tgr) 1770 errors in that one hour, only 240 of them have Wikibase in the stack trace, so not really Wikidata relat...