[05:08:08] <wikibugs>	 10DBA, 10Goal: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1133.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201906250507_marostegui_110176.log`.
[05:13:51] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10Marostegui)
[05:28:34] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1133.eqiad.wmnet'] `  and were **ALL** successful.
[05:32:15] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Goal, 10User-Marostegui: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10Marostegui)
[07:16:35] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1133.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/201906250716_mar...
[07:36:31] <wikibugs>	 10DBA, 10Goal, 10Patch-For-Review: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1133.eqiad.wmnet'] `  and were **ALL** successful.
[07:38:54] <wikibugs>	 10DBA, 10Goal: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment - https://phabricator.wikimedia.org/T220170 (10Marostegui)
[07:38:56] <wikibugs>	 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui)
[07:39:01] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Goal, 10User-Marostegui: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10Marostegui) 05Stalled→03Open Finally db1133 has been installed correctly! Thanks @Cmjohnson for getting it fixed! ` root@db1133:~# megacli -LdPdInfo -...
[07:40:02] <wikibugs>	 10DBA, 10Goal: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment - https://phabricator.wikimedia.org/T220170 (10Marostegui)
[07:40:04] <wikibugs>	 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui)
[07:40:07] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad, 10Goal, 10User-Marostegui: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10Marostegui) 05Open→03Resolved
[07:59:55] <wikibugs>	 10DBA, 10Cognate, 10Growth-Team, 10Language-Team, and 2 others: Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC - https://phabricator.wikimedia.org/T226358 (10Ladsgroup) A new thing that also gets affected is url shortener. FYI.
[08:07:06] <wikibugs>	 10DBA, 10Cognate, 10Growth-Team, 10Language-Team, and 2 others: Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC - https://phabricator.wikimedia.org/T226358 (10Marostegui) Thanks @Ladsgroup! We have always talked about documenting who and which teams to tag when planning x1 switchovers, so I have...
[08:07:24] <marostegui>	 I have finally started:  https://wikitech.wikimedia.org/wiki/MariaDB#Special_section:_x1_master_switchover 
[08:07:30] <marostegui>	 Better than nothing :)
[08:10:20] <jynus>	 cool thanks
[08:10:51] <marostegui>	 I will also add the steps (as they are different from a normal section) once I have them on our etherpad
[08:16:46] <jynus>	 I will put some work on documenting scripts on production
[08:17:00] <jynus>	 and on 10.3 / buster
[08:17:02] <marostegui>	 thanks!
[08:17:03] <marostegui>	 :)
[08:17:27] <jynus>	 also testing the WMFReplication.move()
[08:18:09] <marostegui>	 by the way, the backups goal is finally considered done, right?
[08:18:17] <jynus>	 yes
[08:18:23] <marostegui>	 excellent!
[08:19:52] <jynus>	 but no one's ever really gone! https://youtu.be/pkqRqDI21lg?t=9
[08:20:12] <marostegui>	 hahahah
[08:20:28] <marostegui>	 Is that you holding bacula's power cable?
[08:24:35] <jynus>	 did you edit the backup db recently?
[08:24:59] <marostegui>	 nope
[08:25:05] <marostegui>	 just section_instances table
[08:25:21] <jynus>	 because aparrently unlike what I said about backups yesterday
[08:25:29] <jynus>	 they all were successful, even its metadata
[08:25:39] <jynus>	 s3 was just slow
[08:26:02] <marostegui>	 coooooooool!!
[08:26:11] <jynus>	 took under 5 hours
[08:26:30] <jynus>	 so 0% failure rate in the last 3 runs or so
[08:26:55] <marostegui>	 that's great
[08:26:58] <marostegui>	 congratulations :)
[08:29:44] <jynus>	 https://twitter.com/binlogic/status/1142068667481505792
[08:30:11] <marostegui>	 nice
[08:44:51] <wikibugs>	 10DBA, 10Cognate, 10Growth-Team, 10Language-Team, and 3 others: Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC - https://phabricator.wikimedia.org/T226358 (10Ladsgroup) >>! In T226358#5281302, @Marostegui wrote: > Thanks @Ladsgroup! > We have always talked about documenting who and which teams...
[08:45:44] <wikibugs>	 10DBA, 10Cognate, 10Growth-Team, 10Language-Team, and 3 others: Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC - https://phabricator.wikimedia.org/T226358 (10Marostegui) Thank you! :)
[08:58:57] <wikibugs>	 10DBA, 10Epic: Improve regular production database backups handling - https://phabricator.wikimedia.org/T138562 (10jcrespo)
[09:00:57] <wikibugs>	 10DBA, 10Operations, 10Traffic, 10Patch-For-Review: Framework to transfer files over the LAN - https://phabricator.wikimedia.org/T156462 (10jcrespo) transfer.py was modified to add hot mysql backup taking and compression/decompression handling for provisioning.  It is still a bit of a clunky mess, and it w...
[09:19:56] <wikibugs>	 10DBA, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Marostegui) I have finished deploying the last key change. I did it in small batches during a few hours: https://grafa...
[10:05:17] <wikibugs>	 10DBA, 10Cognate, 10Growth-Team, 10Language-Team, and 3 others: Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC - https://phabricator.wikimedia.org/T226358 (10Tgr) ` tgr@stat1006:~$ analytics-mysql enwiki --use-x1  mysql:research@dbstore1005.eqiad.wmnet [enwiki]> select distinct table_name from...
[10:08:01] <wikibugs>	 10DBA, 10Cognate, 10ContentTranslation, 10Growth-Team, and 9 others: Failover x1 master: db1069 to db1120 3rd July at 06:00 UTC - https://phabricator.wikimedia.org/T226358 (10Marostegui) Thanks a lot @Tgr I will tag those (better to tag them and they can remove themselves if it no longer applies) and updat...
[12:04:15] <jynus>	 there is a problem with TokuDB and buster: https://jira.mariadb.org/browse/MDEV-15034
[12:04:21] <jynus>	 for now I will disable TokuDB
[12:42:07] <marostegui>	 yeah, eventlogging hosts will not be upgraded anyways, hopefully they'll be removed soon
[12:43:44] <jynus>	 or we could move them to Rocksdb if necessary
[12:43:53] <jynus>	 which is now included
[12:43:55] <moritzm>	 is that also an issue that affects mariadb as packaged in Debian?
[12:44:22] <jynus>	 I am unsure, there is a workaround from percona
[12:44:55] <jynus>	 but I usually only use upstream because debian does too much common denominator
[12:51:38] <moritzm>	 ack
[12:53:21] <jynus>	 funnily there is a tokudb related patch on debian
[12:53:26] <jynus>	 but that would not work at all
[12:58:12] <jynus>	 there is a "* mariadb-plugin-tokudb: Properly generate the libjemalloc dependency" Changelog, but I would need to see the diff
[12:59:11] <jynus>	 oh, paravoid is the maintainer of jemalloc, so it is all his fault :-D
[12:59:39] <paravoid>	 clearly :P
[12:59:53] <jynus>	 paravoid: quick question, do you know where I can see the code tracking for a package?
[13:00:01] <jynus>	 the debian code tracking?
[13:00:32] <paravoid>	 "apt showsrc <package> | grep ^Vcs"
[13:00:36] <jynus>	 thanks
[13:00:52] <paravoid>	 or "apt install devscripts; man debcheckout"
[13:02:46] <paravoid>	 salsa.debian.org is usually a good place to search for packaging, but sometimes people use something else
[13:03:11] <paravoid>	 but then they ought to document it in the Vcs-Git header
[13:03:27] <paravoid>	 there is a small minority of strugglers that don't use git but something else like svn, or no VCS at all, sadly :)
[13:03:29] <jynus>	 oh, maybe mysql-5.7 is back!
[13:04:00] <paravoid>	 https://upsilon.cc/~zack/stuff/vcs-usage/
[13:04:11] <jynus>	 thanks, don't worry, I will find it
[13:04:12] <paravoid>	 https://qa.debian.org/cgi-bin/vcswatch
[13:04:36] <paravoid>	 I don't think Debian has oracle mysql at all anymore
[13:04:46] <jynus>	 I saw an ACCEPTED on unstable
[13:04:47] <moritzm>	 it does, but blocked out of stable
[13:04:58] <moritzm>	 https://packages.qa.debian.org/m/mysql-5.7.html
[13:04:59] <paravoid>	 oh really
[13:05:01] <paravoid>	 TIL
[13:05:21] <jynus>	 I had seen movement of renames of mysql -> mariadb
[13:05:34] <jynus>	 so it was only a suspicion before
[13:05:52] <moritzm>	 but I don't thik anyone is working on mysql 8, so 5.7 is probably the end to that story
[13:06:18] <paravoid>	 huh, and that's not even deliberate, it's just because of an RC bug
[13:07:36] <paravoid>	 hah, the uploader for that has an @oracle.com email address
[13:08:03] <moritzm>	 there's a bigger hammer applied there: https://release.debian.org/britney/hints/pochu :-)
[13:10:40] <paravoid>	 oh well
[13:15:41] <jynus>	 just to be clear how much we will cry if TokuDB doesn't make the cut: https://phabricator.wikimedia.org/T109069
[13:16:29] <jynus>	 which is probably somewhat abandonded by Percona too
[13:20:13] <jynus>	 wait I did nothing and compilation migrated to openssl 1.1 on its own, why?
[13:21:05] <wikibugs>	 10DBA, 10Operations, 10Patch-For-Review, 10codfw-rollout: [RFC] improve parsercache replication and sharding handling - https://phabricator.wikimedia.org/T133523 (10Marostegui)
[13:21:07] <wikibugs>	 10DBA, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Marostegui) 05Open→03Resolved
[13:22:24] <jynus>	 I think I know why, there is no libssl1.0 anymore
[13:22:49] <jynus>	 (which is exactly what I wanted)
[13:25:37] <moritzm>	 yeah, on buster openssl 1.0 is gone
[13:26:09] <moritzm>	 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923491
[13:26:31] <jynus>	 stop appearing on my bugs!
[13:26:35] <jynus>	 XD
[13:27:06] <wikibugs>	 10DBA, 10MediaWiki-API, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Waiting for Review), and 2 others: Certain ApiQueryRecentChanges::run api query is too slow, slowing down dewiki - https://phabricator.wikimedia.org/T149077 (10WDoranWMF)
[13:28:36] <paravoid>	 hahaha
[13:29:52] <paravoid>	 I was telling about our TokuDB deployment to a DBA candidate :P
[13:30:01] <paravoid>	 they were asking how much tolerance there is to trying new stuff :)
[13:30:10] <paravoid>	 "it depends"
[13:30:39] <jynus>	 I didn't get that, like recently?
[13:30:43] <paravoid>	 yeah, last week?
[13:31:09] <jynus>	 and telling like our experience (e.g. that ticket?)
[13:31:10] <marostegui>	 paravoid: just send them https://bugs.launchpad.net/percona-server/+bug/1621852
[13:31:25] <paravoid>	 was explaining that trying new things is part of the game, but we also need to maintain some balance etc.
[13:31:54] <paravoid>	 otherwise we may end up with having a new but unstable piece of tech in prod
[13:31:59] <jynus>	 and he was proposing to move faster or the opposite?=
[13:32:12] <jynus>	 *he/she
[13:32:13] <paravoid>	 and then spend a lot of time to keep it stable and/or migrate away from it *cough*
[13:32:19] <paravoid>	 they were not proposing anything, just asking me
[13:32:30] <jynus>	 sure, I just didn't get the context
[13:32:48] <jynus>	 indeed it is a balance
[13:33:04] <jynus>	 but on the other side, there are things that until you truy
[13:33:20] <jynus>	 you only hear a lot of people commenting on things they dfon't really use
[13:33:41] <jynus>	 "this technology is wonderful"--everybody says
[13:33:54] <jynus>	 then you try it and has huge gaps nobody speaks about
[13:34:55] <jynus>	 it would be great to have more resources for more trials
[13:35:10] <jynus>	 the funny thing is, if you ask people with 1 or 2 servers, we are behind on many things
[13:35:17] <paravoid>	 that bug report about tokudb+jemalloc sounds like a broken config on fedora's end frankly
[13:35:38] <jynus>	 but if you ask people with many we are in some cases ahead of the curve
[13:35:48] <paravoid>	 ha, https://jira.percona.com/browse/PS-4393 seems to confirm that, there is a user reporting that it works fine with Ubuntu 18.10 (i.e. Debian's) packaging
[13:35:50] <jynus>	 paravoid: the bug is legitimate
[13:36:05] <paravoid>	 have you reproduced it in buster?
[13:36:05] <jynus>	 tokudb doesn't support jemalloc5
[13:36:08] <jynus>	 yes
[13:36:11] <jynus>	 percona says it
[13:36:22] <jynus>	 the solution is to not use jemalloc for tokudb
[13:36:24] <paravoid>	 what was actually the failure?
[13:36:55] <jynus>	 same, "test not compatible with jemalloc > 4" (paraphrasing)
[13:37:20] <jynus>	 mysql core works wekk
[13:37:23] <paravoid>	 what does that mean?
[13:37:50] <jynus>	 it was explained on the bugh
[13:38:29] <jynus>	 paravoid: https://jira.percona.com/browse/PS-4393
[13:38:55] <paravoid>	 I read that, it says nothing specific
[13:39:09] <jynus>	 yes just that they do not support it
[13:39:24] <jynus>	 is api normally stable then?
[13:39:27] <paravoid>	 yes it is
[13:39:45] <jynus>	 then they said somewhere else they (percona) did something weird
[13:39:59] <paravoid>	 I think it's more likely fedora is not configuring their jemalloc properly
[13:40:10] <jynus>	 well, I am hitting the same error
[13:40:13] <jynus>	 on buster
[13:40:18] <paravoid>	 ok, how?
[13:40:20] <jynus>	 but everything is a user error
[13:40:30] <jynus>	 download the source, see the check failing :-D
[13:40:52] <jynus>	 honestly, I don't care much
[13:41:02] <jynus>	 because it is a good excuse to disable tokudb
[13:41:06] <paravoid>	 mariadb 10.2?
[13:41:09] <jynus>	 we don't want it
[13:41:11] <jynus>	 10.3
[13:41:31] <jynus>	 please don't spend time on it
[13:41:55] <jynus>	 the only patch I see on debian
[13:42:27] <jynus>	 is tokudb-jibjemalloc.path
[13:43:11] <jynus>	 "RPM building seems to do some trickery instead of linking with jemalloc."
[13:43:20] <paravoid>	 Debian's mariadb-plugin-tokudb seems to link against libjemalloc2
[13:43:25] <paravoid>	 version 1:10.3.16-1
[13:43:43] <jynus>	 ?
[13:43:59] <paravoid>	 what?
[13:44:11] <jynus>	 nope, it used to be compiled with 4
[13:44:22] <jynus>	 upsteam source at leasy
[13:44:33] <paravoid>	 Debian's mariadb 10.3.16 package in buster has a binary package for the tokudb plugin, which links against jemalloc 5
[13:45:10] <jynus>	 so it is a mariadb bug
[13:45:22] <paravoid>	 I don't know :)
[13:45:31] <jynus>	 but linked doesn't mean it has been used
[13:45:33] <paravoid>	 is there a build log with what actually fails?
[13:45:45] <jynus>	 yes, cmake fails, not the build
[13:46:12] <jynus>	 but again, I don't care, I prefer to get rid of tokudb
[13:46:18] <jynus>	 which I did
[13:46:19] <paravoid>	 can you show me a build log?
[13:46:22] <jynus>	 sure
[13:46:52] <moritzm>	 stop ruining his technical excuse to disable tokudb :-)
[13:47:01] <paravoid>	 :D
[13:47:39] <jynus>	 I have to configure again, as I build the package already
[13:48:36] <paravoid>	 was the error:
[13:48:38] <paravoid>	   MESSAGE(WARNING "TokuDB is enabled, but jemalloc is not. This configuration is not supported")
[13:48:41] <paravoid>	 or
[13:48:43] <jynus>	 no
[13:48:44] <paravoid>	     MESSAGE(FATAL_ERROR "static jemalloc_pic.a can only be used up to jemalloc 4")
[13:48:50] <jynus>	 yes, that one
[13:48:58] <paravoid>	 the second?
[13:49:02] <jynus>	 I am getting it for you, just configuring takes some time
[13:49:35] <jynus>	 happy? https://phabricator.wikimedia.org/P8655
[13:50:15] <paravoid>	 what were the arguments to ./configure?
[13:50:43] <jynus>	 I think I enabled tokudb with -DBUILD_CONFIG=mysql_release
[13:51:32] <jynus>	 which is very similar to what debian does
[13:51:46] <paravoid>	 what is the full argument list of ./configure? :)
[13:51:52] <jynus>	 cmake . -DBUILD_CONFIG=mysql_release -DWITH_SSL=system -DPLUGIN_MROONGA=NO -DPLUGIN_OQGRAPH=NO -DPLUGIN_AWS_KEY_MANAGEMENT=NO
[13:52:07] <paravoid>	 ok
[13:52:28] <paravoid>	 and you have libjemalloc-dev 5.1.0-3 installed?
[13:52:57] <jynus>	 libjemalloc-dev/testing,now 5.1.0-3 amd64 [installed]
[13:53:13] <jynus>	 if not, mysql itself wouldn't compile
[13:53:44] <paravoid>	 why wouldn't it? it's not using jemalloc by default I think?
[13:53:59] <jynus>	 with release config yes
[13:54:14] <jynus>	 honestly, I think debian may just have applied: https://sources.debian.org/patches/mariadb-10.3/1:10.3.15-1/tokudb-libjemalloc.patch/
[13:54:18] <jynus>	 and forgotten
[13:54:25] <paravoid>	 no that's not it
[13:54:37] <paravoid>	 I don't think it is at least
[13:54:52] <jynus>	 I didn't think either
[13:55:00] <paravoid>	 -- Looking for malloc_stats_print in jemalloc_pic
[13:55:00] <paravoid>	 	-- Looking for malloc_stats_print in jemalloc_pic - found
[13:55:27] <paravoid>	 this means that this is configured to link with jemalloc statically
[13:55:57] <paravoid>	 rather than dynamically, i.e. with a runtime dependency against libjemalloc2
[13:56:12] <paravoid>	 (which is the part that doesn't work with tokudb + jemalloc 5)
[13:56:20] <jynus>	 more reasons to disable it!
[13:56:24] <paravoid>	 that was from your build log
[13:56:34] <paravoid>	 and is a bug, we should not do that
[13:56:40] <paravoid>	 Debian's build log reads instead:
[13:56:40] <paravoid>	 -- Looking for malloc_stats_print in jemalloc
[13:56:41] <paravoid>	 -- Looking for malloc_stats_print in jemalloc - found
[13:56:43] <paravoid>	 which is the right behavior
[13:56:46] <paravoid>	 and that is independent of tokudb
[13:57:01] <jynus>	 I don't think that is true
[13:57:15] <paravoid>	     IF(WITH_JEMALLOC STREQUAL "static")
[13:57:15] <paravoid>	       SET(libname jemalloc_pic)
[13:57:15] <jynus>	 as in, it is indeed bad
[13:57:18] <paravoid>	     ELSE()
[13:57:18] <paravoid>	       SET(libname jemalloc c)
[13:57:31] <paravoid>	 that's from cmake/jemalloc.cmake 
[13:58:03] <paravoid>	 the question is why $ grep WITH_JEMALLOC cmake/build_configurations/mysql_release.cmake 
[13:58:06] <paravoid>	   SET(WITH_JEMALLOC static CACHE STRING "")
[13:58:08] <paravoid>	 there we go
[13:58:11] <paravoid>	 responded to the question I was to ask
[13:58:28] <jynus>	 which was what I said at the beginning
[13:58:35] <paravoid>	 try
[13:58:38] <paravoid>	 -DWITH_JEMALLOC=yes
[13:58:44] <jynus>	 oh, I see
[13:58:49] <jynus>	 it is the implicit
[13:59:07] <paravoid>	 it is implicitly set to "static", not "yes"/"system"/"auto" (all equivalent)
[13:59:27] <jynus>	 that must be new on 10.3
[13:59:37] <paravoid>	 probably a bug in Debian's packaging too
[13:59:55] <paravoid>	 hm, no, that can't be it -- I wonder how
[14:00:39] <jynus>	 it worked
[14:00:48] <paravoid>	 cool
[14:01:09] <paravoid>	 "dpkg --info mariadb*deb |grep ^Depends" should list a libjemalloc2 dependency now
[14:01:32] <jynus>	 hey, we need 2 hours to compile mysql!
[14:01:46] <jynus>	 so be patient!
[14:01:52] <paravoid>	 have you tried
[14:01:53] <paravoid>	 export DEB_BUILD_OPTIONS="parallel=8"
[14:01:54] <paravoid>	 already?
[14:05:06] <jynus>	 still not instant :-D
[14:05:45] <paravoid>	 seems like you'd need to manually add code so that gets translated to cmake flags, search for DEB_BUILD_OPTIONS in Debian's d/rules
[14:06:16] <jynus>	 I mean, it is using now all cores, but it still takes some time
[14:06:21] <paravoid>	 ah ok
[14:06:38] <jynus>	 thanks, I thought cmake was parallel by default
[14:07:18] <jynus>	 I still don't like having tokudb :-(
[14:07:26] <paravoid>	 well ok, that's not up to me :)
[14:08:14] <paravoid>	 both MDEV-15034 and PS-4393 are full of misunderstandings :(
[14:08:49] <jynus>	 I don't think that much as the build options are not clear
[14:09:17] <jynus>	 also I can swear jemalloc used to be the default option
[14:09:26] <jynus>	 dynamically
[14:10:00] <paravoid>	 dunno what to tell you :)
[14:10:03] <jynus>	 *could
[14:10:51] <paravoid>	 took us what, 10 minutes to find a fix after I saw the build log?
[14:11:07] <paravoid>	 anyway
[14:12:22] <jynus>	 I still don't see jemalloc dinamically linked
[14:13:13] <jynus>	 https://phabricator.wikimedia.org/P8655#52019
[14:13:36] <paravoid>	 did the build log say "- Looking for malloc_stats_print in jemalloc"
[14:13:40] <paravoid>	 or jemalloc_pic?
[14:14:38] <jynus>	 HAVE_JEMALLOC_IN_jemalloc:INTERNAL=1
[14:15:05] <jynus>	 I think it is not yes
[14:15:11] <jynus>	 but system or something else
[14:15:16] <paravoid>	 no
[14:16:39] <paravoid>	 where was that line from?
[14:17:04] <jynus>	 CMakeCache.txt
[14:17:10] <paravoid>	 grep for MALLOC_LIBRARY
[14:17:24] <paravoid>	 both CMakeCache.txt, and a config.h that should be there somewhere
[14:21:03] <paravoid>	 hmm, it may be "system jemalloc"
[14:21:22] <jynus>	 ah!
[14:21:32] <jynus>	 which may produce the same error
[14:21:34] <paravoid>	 no
[14:21:36] <paravoid>	 not error
[14:21:38] <paravoid>	 what is it?
[14:21:43] <paravoid>	 can you check CMakeCache.txt?
[14:22:07] <jynus>	 I pasted it to you already but:
[14:22:14] <jynus>	 https://phabricator.wikimedia.org/P8655#52020
[14:22:36] <paravoid>	 so no MALLOC_LIBRARY?
[14:22:46] <paravoid>	 check config.h
[14:23:49] <jynus>	 #define MALLOC_LIBRARY "system"
[14:23:57] <paravoid>	 huh
[14:25:56] <paravoid>	 can you paste your entire build log?
[14:26:18] <paravoid>	 I'm interested in the lines that start with "Looking for malloc_stats_print"
[14:26:26] <wikibugs>	 10DBA, 10Goal: Productionize db11[26-38] - https://phabricator.wikimedia.org/T222682 (10Marostegui)
[14:28:07] <paravoid>	 hmmmmmm
[14:28:12] <paravoid>	 that could be what Debian's patch is about
[14:28:22] <paravoid>	 the IF(... DEB) UNSET(LIBJEMALLOC)
[14:28:29] <jynus>	 wait
[14:28:33] <paravoid>	 yeah, I bet this is it
[14:28:39] <jynus>	 I was ldd'ing the wrong binary
[14:28:45] <paravoid>	 so try applying tokudb-libjemalloc.patch
[14:29:12] <paravoid>	 i.e. both the -DWITH_JEMALLOC=yes + tokudb-libjemalloc.patch
[14:29:27] <jynus>	 but still I get https://phabricator.wikimedia.org/P8655#52021
[14:30:09] <paravoid>	 see above :)
[14:30:12] <jynus>	 let me try first a jemalloc=yes + no toku just in case
[14:30:21] <paravoid>	 that will probably also work
[14:30:30] <paravoid>	 because the
[14:30:30] <paravoid>	     UNSET(LIBJEMALLOC)
[14:30:31] <jynus>	 so toku is bad in many ways
[14:30:34] <paravoid>	 is under storage/tokudb/CMakeLists.txt
[14:30:35] <paravoid>	 lol
[14:30:36] <jynus>	 this is only one of them
[14:31:52] <paravoid>	 sounds like you've already made up your mind about that :P
[14:32:11] <jynus>	 so it is not a petty revenge
[14:32:14] <jynus>	 it has lots and lots of code
[14:32:28] <paravoid>	 I don't really care about tokudb :)
[14:32:32] <jynus>	 with more opportunity for bugs, increase on package size, etc.
[14:32:33] <paravoid>	 (either way)
[14:32:52] <jynus>	 and it has specials stuff like this config (probably)
[14:32:59] <jynus>	 because it requires special version of stuff
[14:33:02] <paravoid>	 nah, this is just weird build system in general
[14:33:09] <jynus>	 yeah, indeed
[14:33:20] <jynus>	 but mostly due to mixing lots of plugins
[14:33:38] <paravoid>	 there are basically 3 ways to do jemalloc with (core) mariadb, not counting tokudb's special stuff
[14:33:41] <jynus>	 we can always pluging it later and compile it separatelly
[14:33:45] <paravoid>	 a) link with jemalloc statically (default)
[14:33:57] <jynus>	 oh, we had some fun with libssl too :-D
[14:33:58] <paravoid>	 b) link with jemalloc dynamically, but with a hard dependency
[14:34:29] <paravoid>	 c) do neither, but link dynamically at runtime by passing --with-malloc-lib=jemalloc
[14:34:51] <jynus>	 because the plugins sometimes use external libs too
[14:34:57] <jynus>	 and they do many hacks
[14:35:08] <paravoid>	 by passing --with-malloc-lib=jemalloc to mysqld_safe, that is
[14:35:13] <paravoid>	 all of that has nothing to do with plugins
[14:35:21] <jynus>	 mysql_safe doesn't exist any more
[14:35:25] <jynus>	 at least not for us
[14:36:04] <paravoid>	 systemd equivalent is to do Environment=LD_PRELOAD=...
[14:37:03] <jynus>	 I don't think that worked, but I am compiling anyway
[14:37:10] <jynus>	 I think the unset may be the cause
[14:37:18] <jynus>	 the root one
[14:37:36] <paravoid>	 the UNSET is part of the tokudb cmakefile
[14:37:38] <jynus>	 and the test runs even with tokudb disabled
[14:37:43] <paravoid>	 ah, maybe
[14:37:43] <jynus>	 don't ask me why
[14:37:46] <jynus>	 :-D
[14:37:59] <paravoid>	 no, it shouldn't
[14:38:04] <jynus>	 paravoid: welcome to mysql development 101!
[14:38:14] <paravoid>	 no, I mean there is code that would prevent that
[14:38:14] <jynus>	 it is bad indeed
[14:38:18] <jynus>	 I know
[14:38:27] <jynus>	 I am stating the state of the build
[14:38:37] <paravoid>	 what did you pass to disable TokuDB?
[14:38:59] <jynus>	 -DPLUGIN_TOKUDB=NO
[14:39:56] <paravoid>	 Debian seems to do -DWITHOUT_TOKUDB=true instead
[14:39:59] <paravoid>	 (on non-amd64)
[14:40:05] <jynus>	 interesting
[14:40:19] <jynus>	 mine was the traditional one
[14:40:47] <jynus>	 "e.g. this is how you enable/disable/statically link plugins"
[14:43:17] <jynus>	 and it works as in, it doesn't try to compile it anymore
[14:44:55] <jynus>	 I know I am sure why it used to compile dynamically
[14:45:03] <jynus>	 because I did it but messed up my deps
[14:45:25] <jynus>	 and it basically didn't start, and to fix it manually on puppet for 1 realease
[14:46:17] <paravoid>	 I don't know what that means but OK
[14:46:21] <paravoid>	 don't have to :)
[14:46:22] <jynus>	 he he
[14:46:38] <jynus>	 don't worry, I will figure this out one way or another
[14:47:11] <jynus>	 I wasn't preciselly going to deploy this to production any tiome soon
[14:47:29] <paravoid>	 I'd apply tokudb-libjemalloc.patch anyway
[14:47:35] <jynus>	 yes, that was next
[14:47:48] <paravoid>	 and pass -DWITH_JEMALLOC=yes if you'd like to run this with jemalloc
[14:48:02] <paravoid>	 and disable tokudb if you so desire :)
[14:50:20] <paravoid>	 grep MALLOC_LIBRARY config.h should (probably!) be "system jemalloc" at the end of all this
[14:50:55] <jynus>	 that is the plan
[14:51:25] <paravoid>	 gotta say, kind of a shame we're reinventing a mariadb package from scratch
[14:51:36] <paravoid>	 rather than taking what's in Debian and removing the stuff that we don't like
[14:51:39] <jynus>	 paravoid: there is now a core package we could use
[14:51:55] <paravoid>	 (like toku! :P)
[14:51:55] <jynus>	 without the mariadb-server horrible stuff
[14:52:10] <jynus>	 but they compile with yassl or whatever is the nee thing
[14:52:14] <jynus>	 *new
[14:52:34] <paravoid>	 gnutls looks like
[14:53:34] <jynus>	 but they don't build anyway mysql and other stuff
[14:53:55] <jynus>	 nor the package allows multiple versions at the same time
[14:55:40] <jynus>	 (maybe we should migrate mysql to kubernetes!)
[14:56:33] <jynus>	 just to be clear- we can and do use the debian package, our puppet accounts for taht option
[16:56:54] <wikibugs>	 10DBA, 10MediaWiki-Database, 10Operations, 10Patch-For-Review: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10jcrespo) [[ https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-dc=eqiad%20promet...
[17:08:16] <wikibugs>	 10DBA, 10MediaWiki-Database, 10Operations, 10Patch-For-Review: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10Marostegui) >>! In T193224#5283230, @jcrespo wrote: > [[ https://grafana.wikimedia.org/d/...
[17:10:58] <wikibugs>	 10DBA, 10MediaWiki-Database, 10Operations, 10Patch-For-Review: Evaluate and decide the future of relational datastore at WMF after the upgrade of MariaDB 10.1 is finished - https://phabricator.wikimedia.org/T193224 (10jcrespo) > Nice! Is it replicating from the master?  It is, although tendril doesn't seem...
[18:06:22] <wikibugs>	 10DBA, 10MediaWiki-extensions-Babel, 10translatewiki.net: babel database doesn't support language codes longer than 10 characters (e.g. de-x-formal) - https://phabricator.wikimedia.org/T226546 (10Legoktm) Hi #DBA, input requested please :)  When I created the `babel` database table nearly 3 years ago, the lo...
[18:51:00] <wikibugs>	 10DBA, 10MediaWiki-extensions-Babel, 10translatewiki.net: babel database doesn't support language codes longer than 10 characters (e.g. de-x-formal) - https://phabricator.wikimedia.org/T226546 (10jcrespo) Is it only on the wikis you pasted, or could it be on more?  > If we made the column varchar(30), would...
[19:52:37] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1072 - https://phabricator.wikimedia.org/T226569 (10Marostegui) a:03Cmjohnson Can we get this disk replaced  - this is m3 master. Thanks!
[21:52:52] <wikibugs>	 10DBA, 10MediaWiki-extensions-Babel, 10translatewiki.net: babel database doesn't support language codes longer than 10 characters (e.g. de-x-formal) - https://phabricator.wikimedia.org/T226546 (10Reedy) >>! In T226546#5283845, @jcrespo wrote: > Is it only on the wikis you pasted, or could it be on more?  Nop...