[00:03:47] (CR) Ejegg: [C: +2] Add Wikimedia Endowment logos [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483632 (https://phabricator.wikimedia.org/T203603) (owner: Ejegg) [00:03:50] (CR) Ejegg: [C: +2] Replace Endowment logos with transparent versions [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483633 (https://phabricator.wikimedia.org/T203603) (owner: Ejegg) [00:04:04] (CR) Ejegg: [C: +2] variant config for full redirect [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483634 (owner: Ejegg) [00:07:40] (CR) Ejegg: [C: +2] Merge branch 'REL1_31' into fundraising/REL1_31 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483626 (owner: Ejegg) [00:11:24] (Merged) jenkins-bot: Merge branch 'REL1_31' into fundraising/REL1_31 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483626 (owner: Ejegg) [00:11:36] (Merged) jenkins-bot: Add Wikimedia Endowment logos [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483632 (https://phabricator.wikimedia.org/T203603) (owner: Ejegg) [00:11:42] (Merged) jenkins-bot: Replace Endowment logos with transparent versions [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483633 (https://phabricator.wikimedia.org/T203603) (owner: Ejegg) [00:12:30] (Merged) jenkins-bot: variant config for full redirect [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483634 (owner: Ejegg) [00:14:20] (PS1) Ejegg: Update DonationInterface and FundraisingEmailUnsubscribe [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483641 [00:50:18] (CR) Ejegg: [C: +2] Update DonationInterface and FundraisingEmailUnsubscribe [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483641 (owner: Ejegg) [00:53:21] (Merged) jenkins-bot: Update DonationInterface and FundraisingEmailUnsubscribe [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/483641 (owner: Ejegg) [06:57:13] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [07:02:13] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-client] [07:07:13] RECOVERY - check_puppetrun on frdb2001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:54:39] (CR) AndyRussG: "Nice!!! Works as advertised (though see inline comment for a detail about edits via the Translate extension)." (3 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/481099 (https://phabricator.wikimedia.org/T209795) (owner: Ejegg) [17:25:48] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Incorrect protection reason for non-source-language version of translatable banner messages - https://phabricator.wikimedia.org/T213565 (AndyRussG) [17:26:48] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Incorrect protection reason for non-source-language version of translatable banner messages - https://phabricator.wikimedia.org/T213565 (AndyRussG) [17:29:49] fr-tech I'll be a few min late to standup, sorry! [17:30:06] ejegg|afk: k :) [19:58:01] Fundraising Sprint A 2019, Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Renew Ingenico production API key - https://phabricator.wikimedia.org/T213216 (Ejegg) Open→Resolved p:Triage→High a:Ejegg [20:23:32] Fundraising Sprint A 2019, Fundraising Sprint Window dressing is mostly olive oil, Fundraising Sprint XML ate my homework, Fundraising-Backlog, and 2 others: Ingenico refunds in mismatched currency - https://phabricator.wikimedia.org/T210473 (Ejegg) [20:23:35] Fundraising Sprint A 2019, Fundraising Sprint XML ate my homework, Fundraising-Backlog, Fr-Ingenico-integration_2017-18, Patch-For-Review: currency mismatch between Ingenico Connect & Civi - https://phabricator.wikimedia.org/T211024 (Ejegg) Open→Resolved Verified with payment 40009280... [20:35:28] Fundraising Sprint A 2019, Fundraising Sprint XML ate my homework, Fundraising-Backlog, Fr-Ingenico-integration_2017-18, Patch-For-Review: currency mismatch between Ingenico Connect & Civi - https://phabricator.wikimedia.org/T211024 (Ejegg) Oops @Eileenmcnaughton , they DO have the same root... [20:39:43] @Jeff_Green, got a minute to check in on master-slave setup? I don't have any specific q's right now - just want to see if these bits are making sense [20:40:00] totten: sure [20:40:19] i'm keeping most of configs/commands in a small git repo [20:40:37] there's three bits for my.cnf: [20:40:56] common stuff -- mostly performance relatd -- https://github.com/totten/rundb/blob/master/templates/common/my.cnf [20:41:06] looking [20:41:11] includes two sync bits: binlog_format=row and sync_binlog=1 [20:41:40] and then smaller files for the master-specific and slave-specific bits [20:41:42] https://github.com/totten/rundb/blob/master/templates/master/my.cnf [20:41:45] https://github.com/totten/rundb/blob/master/templates/slave/my.cnf [20:41:57] ok [20:42:36] it's been a long time since I set this up from scratch, so I need to review a bit to remember why we use the settings we do [20:43:30] to setup the sync relations, there were maybe 5-12 commands needed; since i go between a couple laptops regularly, i find it easier to put them in a script: [20:43:30] we use the same sync_binlog and binlog_format [20:43:31] https://github.com/totten/rundb/blob/master/bin/init-repl#L33 [20:44:05] yay! re: binlog settings [20:45:10] we explicitly set log_bin and log_bin_index, let's see if that's necessary or if the default is good [20:45:33] (reads about those settings) [20:45:38] have you started the master with this config yet? [20:46:29] I ~think~ by setting log_bin you tell the master to actually enable binlogging [20:46:33] the master one has `log-bin` in https://github.com/totten/rundb/blob/master/templates/slave/my.cnf#L17 -- but doesn't hvave log_bin_index [20:47:06] I think that's fine [20:47:29] yes, i've started it -- the replication seemed to work correctly when manually a small db and a small table [20:47:52] yeah? then I think you're in good shape [20:48:21] we don't do very much tweaking from defaults re. replication [20:48:49] in addiition to what you have we do... [20:48:58] also, i did a "mysqldump" from my regular DB and piped it to "mysql" on the master DB CLI -- the cmd seemed to work. it didn't complain. [20:48:59] expire_logs_days = 10 [20:49:21] ("mysqldump" from my regular *civi development* db) [20:49:30] great [20:49:54] "expire_logs_days = 10" ==> that's probably common to master+slave? or just master? [20:50:18] that's on the master, not saying that makes sense for you specifically [20:50:59] we set it there as a balance between being able to start a slave from a dump, and not overrunning disk on the master [20:52:03] on the slaves we enable "read_only" for safety, this could be handy for testing I guess [20:52:05] i'll add it to master then. (seems easy and bring its closer to real config) [20:52:46] well, we need to be able to create temp tables (advanced querying) on the slave, so "read_only" might be too tight? [20:53:03] true temp tables are not affected [20:53:34] i.e. the normal session-temporal stuff is still possible, you just can't write the stored databases [20:53:59] ooh. in that case, i might give read_only a trie. (for fake/durable temp tables, it should be doing everything on master anyway.) [20:54:24] yep, and we'd like to keep the slavedb's that way, so if it's a problem this would be a good time to find out [20:55:50] do these GRANT's look fair for a process reading on the slave? https://github.com/totten/rundb/blob/master/bin/init-repl#L61 [20:57:15] looking at the master part, yes but I'm a little confused about @'127.0.0.1' [20:57:51] ordinarily that would be the client's IP the master sees for the replication connection [20:58:12] you're running this on two separate vb instances right? [20:59:47] eh, well, i started thinking about what would be involved with keeping configs up-to-date between two VMs multipled by two laptops -- and decided to try just launching two copies of mysqld on localhost [20:59:50] https://github.com/totten/rundb#rundb-quick-and-dirty-local-dev-wmariadb-master-slave [21:00:05] ahhh ok that makes sense! [21:00:20] then yes, this all looks good to me [21:00:22] o/ [21:00:43] and the fact that it's working is probably the best indicator of all :-P [21:00:45] network lag *can* be worse for row based replication [21:01:06] if you have huge result sets to update [21:01:46] yeah, as you've seen that's not a simple question/answer [21:01:55] highly application specific [21:02:04] yep [21:02:15] that's a really good point re:lag [21:02:20] just thinking of reasons running them on the same box might perform differently [21:02:53] I think you're going to have a hard time generating lag unless you specifically do something to cause it [21:03:36] the organic lag of two colocated vms would also be pretty low, tho, right? [21:04:00] what mechanism would one use to inject/simlate lag? [21:04:05] so far we've seen nothing significant at all since switching to row [21:04:11] yeah [21:04:20] i wouldn't even bother because it's so case speicifc [21:04:23] you could even just stop/start slave [21:04:57] the replication stream is single-threaded, so anything that would tie up that thread causes lag [21:05:02] thinking [21:05:38] nice 19 [21:05:40] yeah, seemed like stop/start'ing the slave (or msater) was a pretty intuitive way to create ucky edge-cases [21:05:43] with row replication, if you had a preposterously large table with no primary key, I think you could concievably have a single row update that takes forever? [21:06:09] "nice 19" => nice :) [21:06:37] hah! caps lock encoded [21:08:29] I guess you could even lock a critical table on the slavedb [21:08:57] does that work with read_only enabled i wonder... [21:09:53] i hvae never succeeded in making realistic lag [21:10:41] open a million temp tables on the slavedb? [21:11:04] i'm guessing you can throttle the interfaces in vb? [21:11:19] we had theories why that destroyed replication performance, but we didn't have any way to test/prove them [21:11:36] that's an interesting idea [21:12:03] although he's running both instances on the host machine, not in vb [21:12:07] the only thing i can think of that makes vms more "realistic" is it uses the kernel networking stack [21:12:20] you could write a small port-redirector (proxy) which randomly calls usleep() :) [21:12:30] nice! [21:12:45] heh [21:12:48] i bet it's been done [21:13:09] must have been done [21:13:20] in fact i bet it's relied on heavily for production at a shockingly large company [21:14:11] is there a network equivalent for 'nice' ? [21:14:57] ionice? [21:15:18] hmmmmm [21:15:20] man tc [21:15:23] i have never used this [21:15:38] me either [21:17:47] i think vmware may have this functionality (if you don't mind spending $200+ on the license) [21:18:06] google also comes up with some tools/cmds for various os's/kernels [21:21:44] So I enabled "read_only" on the slave config and rebuilt the master+slave [21:22:03] a bit surprised that the `root` user was able to insert a row on a replicated table: [21:22:23] (console log coming in a moment) [21:22:39] on the slave? yeah that is a little strange [21:24:50] https://gist.github.com/totten/a0b8681d92fd3a50d04d9b96d7b66a48 [21:26:44] (verified that the master and slave connections were really to the master and slave db's -- e.g. the values of `select @@read_only, @@port` looks right [21:26:58] cool [21:28:39] ah, but it seems to be an exception for the "root" user [21:29:06] right, it makes sense that you would want to be able to intervene as root user I guess [21:30:07] i wonder if the root user can even toggle that setting without restarting the database [21:31:55] * totten needs to remember how to quickly paste blurbs of text into irc [21:32:58] (PS1) Ejegg: Fix recurring records charged as USD [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/483862 (https://phabricator.wikimedia.org/T210473) [21:33:39] anyway, looks like you can change it - https://pastebin.com/EV05j4LG [21:33:56] Fundraising Sprint A 2019, Fundraising Sprint Window dressing is mostly olive oil, Fundraising Sprint XML ate my homework, Fundraising-Backlog, and 3 others: Ingenico refunds in mismatched currency - https://phabricator.wikimedia.org/T210473 (Ejegg) p:Triage→Normal a:Ejegg [21:34:15] good to know [21:38:09] AndyRussG|ish: any priority items to review? [21:57:41] Does it seem alright if I only have the Civi DB replicated? Or should the Drupal DB be replicated as well? [21:59:14] totten: if it's present on the master it should already be replicated, unless you excluded it from replication [22:00:07] ah, it's not. right, now i do a build with usual (non-replicated) setup -- then copy over DB's to the replicated setup [22:01:11] yeah you'll either want both instances to have the same databases in the same state when you first start replication, or you'll need to exclude databases from the replication stream [22:01:12] i can either (1) copy it over as well or (2) change my regular build system to use the replicate dbms [22:01:56] you can probably stop slave, dump on master, restore on slavedb, and start replication [22:11:34] (PS1) QChris: Allow “Gerrit Managers” to import history [wikimedia/fundraising/centralnotice_analytics] (refs/meta/config) - https://gerrit.wikimedia.org/r/483869 [22:11:37] (CR) QChris: [V: +2 C: +2] Allow “Gerrit Managers” to import history [wikimedia/fundraising/centralnotice_analytics] (refs/meta/config) - https://gerrit.wikimedia.org/r/483869 (owner: QChris) [22:11:58] (PS1) QChris: Import done. Revoke import grants [wikimedia/fundraising/centralnotice_analytics] (refs/meta/config) - https://gerrit.wikimedia.org/r/483870 [22:12:01] (CR) QChris: [V: +2 C: +2] Import done. Revoke import grants [wikimedia/fundraising/centralnotice_analytics] (refs/meta/config) - https://gerrit.wikimedia.org/r/483870 (owner: QChris) [22:13:35] ejegg hmmm not really just now, thanks! [22:22:43] @Jeff_Green, so on the Drupal side -- do you just keep the slave around as a possible backup -- or is there some mechanism which distribute SQL queries (some to master, some to slave)? [22:23:54] or maybe it doesn't matter much either way for purposes of hte civi code [22:24:12] we've managed to move some of the longer reporting-type queries off of the master, and I guess there may be some access of the drupal db for those? [22:25:48] that could be. Shouldn't be in a stock config, but it depends on how any custom modules are written. [22:26:27] any thoughts @eileen ^^? [22:54:19] I'm gonna head out, have a good weekend all!