[00:00:31] yep [00:01:54] yep, agreed an am down to restart them if the line stays flat [00:12:18] Fundraising Sprint turtles that are robotic that destroy the whole world with their foot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Coinbase import error - https://phabricator.wikimedia.org/T177806#3681745 (LeanneS) The import now works! Thanks so much. I'll also request that they use the... [00:14:15] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Weird contribution merge in Civi - https://phabricator.wikimedia.org/T178021#3681757 (LeanneS) That's very good to know. I'll let others on the team know to be aware. [00:26:25] cwd I think that is long enough to demonstrate the temp tables are no longer going up [00:34:18] eileen1: \o/ [00:36:35] cwd I found this statement " As I mentioned, each transaction goes into the binlog in the order in which it commits, not in the order in which it is started, so transactions may execute in a different order on the replica. " [00:37:00] I can't think why the drop would commit before the create but I guess somehow it does [00:42:35] eileen1: where did you find that? [00:43:03] https://www.xaprb.com/blog/2007/01/20/how-to-make-mysql-replication-reliable/ [00:45:38] yeah based on that i agree that is seems weird that the drop would come first [00:49:08] I'm feeling optimistic that this will make a difference - that job is fairly low volume so it had not been a focus until your spotting of open temp tables & jeff's digging into the binlog found something that finally looked like a smoking gun [00:50:55] i definitely agree that it would make sense as a culprit...but i've thought that so many times so far :) [00:51:24] either way that value increasing forever can't be good [00:51:51] and without your expertise we would not have had the faintest idea where to look [00:57:07] eileen1: think i should restart the slaves' mysql procs? [00:58:59] cwd yep [01:05:24] eileen1: i think i'll leave 2001 as a control [01:10:14] RECOVERY - check_swap on frdb1002 is OK: SWAP OK - 100% free (7595 MB out of 7629 MB) [02:50:14] PROBLEM - check_disk on alnitak is CRITICAL: DISK CRITICAL - free space: / 749 MB (10% inode=80%): /sys/fs/cgroup 0 MB (100% inode=99%): /dev 32184 MB (99% inode=99%): /run 6438 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32196 MB (100% inode=99%): /run/user 100 MB (100% inode=99%): /boot 182 MB (72% inode=99%): /srv 1305092 MB (26% inode=99%) [02:55:14] PROBLEM - check_disk on alnitak is CRITICAL: DISK CRITICAL - free space: / 721 MB (10% inode=80%): /sys/fs/cgroup 0 MB (100% inode=99%): /dev 32184 MB (99% inode=99%): /run 6438 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32196 MB (100% inode=99%): /run/user 100 MB (100% inode=99%): /boot 182 MB (72% inode=99%): /srv 1305092 MB (26% inode=99%) [03:00:14] PROBLEM - check_disk on alnitak is CRITICAL: DISK CRITICAL - free space: / 694 MB (9% inode=80%): /sys/fs/cgroup 0 MB (100% inode=99%): /dev 32184 MB (99% inode=99%): /run 6438 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32196 MB (100% inode=99%): /run/user 100 MB (100% inode=99%): /boot 182 MB (72% inode=99%): /srv 1304963 MB (26% inode=99%) [03:05:05] PROBLEM - check_disk on alnitak is CRITICAL: DISK CRITICAL - free space: / 666 MB (9% inode=80%): /sys/fs/cgroup 0 MB (100% inode=99%): /dev 32184 MB (99% inode=99%): /run 6438 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32196 MB (100% inode=99%): /run/user 100 MB (100% inode=99%): /boot 182 MB (72% inode=99%): /srv 1304963 MB (26% inode=99%) [03:10:14] PROBLEM - check_disk on alnitak is CRITICAL: DISK CRITICAL - free space: / 639 MB (9% inode=80%): /sys/fs/cgroup 0 MB (100% inode=99%): /dev 32184 MB (99% inode=99%): /run 6438 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32196 MB (100% inode=99%): /run/user 100 MB (100% inode=99%): /boot 182 MB (72% inode=99%): /srv 1304963 MB (26% inode=99%) [03:13:06] my fault [03:15:14] RECOVERY - check_disk on alnitak is OK: DISK OK - free space: / 3385 MB (48% inode=80%): /sys/fs/cgroup 0 MB (100% inode=99%): /dev 32184 MB (99% inode=99%): /run 6438 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32196 MB (100% inode=99%): /run/user 100 MB (100% inode=99%): /boot 182 MB (72% inode=99%): /srv 1304837 MB (26% inode=99%) [06:50:37] (Draft2) Obaid Raza: Setting aliases of some special page names for Urdu language [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/383974 [15:22:58] (PS1) Mepps: Fix test + bug for orphan slayer [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/384065 [15:23:04] (CR) jerkins-bot: [V: -1] Fix test + bug for orphan slayer [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/384065 (owner: Mepps) [15:23:47] (PS2) Mepps: Fix test + bug for orphan slayer [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/384065 [16:23:52] (CR) Jforrester: [C: 2] Setting aliases of some special page names for Urdu language [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/383974 (owner: Obaid Raza) [16:45:56] (Merged) jenkins-bot: Setting aliases of some special page names for Urdu language [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/383974 (owner: Obaid Raza) [17:15:37] (PS1) Mepps: Updated donation interface for orphan_slayer Bug: T172202 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/384077 (https://phabricator.wikimedia.org/T172202) [17:22:40] Fundraising Sprint turtles that are robotic that destroy the whole world with their foot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Coinbase import error - https://phabricator.wikimedia.org/T177806#3670849 (XenoRyet) Open>Resolved [17:25:53] (CR) jerkins-bot: [V: -1] Updated donation interface for orphan_slayer Bug: T172202 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/384077 (https://phabricator.wikimedia.org/T172202) (owner: Mepps) [17:44:51] (PS2) Mepps: Updated donation interface for orphan_slayer and module update Bug: T172202 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/384077 (https://phabricator.wikimedia.org/T172202) [17:47:19] (CR) Mepps: [C: 1] "Mostly looks good! Just see my one question." (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/383949 (https://phabricator.wikimedia.org/T178086) (owner: XenoRyet) [17:48:41] (CR) jerkins-bot: [V: -1] Updated donation interface for orphan_slayer and module update Bug: T172202 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/384077 (https://phabricator.wikimedia.org/T172202) (owner: Mepps) [17:52:53] (PS3) Mepps: Updated donation interface for orphan_slayer and module update Bug: T172202 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/384077 (https://phabricator.wikimedia.org/T172202) [17:55:13] (CR) XenoRyet: Handle additional type of failed recurrance. (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/383949 (https://phabricator.wikimedia.org/T178086) (owner: XenoRyet) [17:55:28] mepps: ^ [17:56:48] (CR) Mepps: [C: 2] "This looks good and like an important fix!" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/383949 (https://phabricator.wikimedia.org/T178086) (owner: XenoRyet) [17:56:53] (CR) jerkins-bot: [V: -1] Updated donation interface for orphan_slayer and module update Bug: T172202 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/384077 (https://phabricator.wikimedia.org/T172202) (owner: Mepps) [17:57:08] Thanks! [17:57:50] (Merged) jenkins-bot: Handle additional type of failed recurrance. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/383949 (https://phabricator.wikimedia.org/T178086) (owner: XenoRyet) [17:58:37] (CR) Mepps: "This needs to wait until the DI fix is merged." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/384077 (https://phabricator.wikimedia.org/T172202) (owner: Mepps) [18:06:33] (CR) Mepps: [C: 2] "Other than white space, which looks to be a separate patch, this looks good!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381499 (owner: Ejegg) [18:10:27] (Merged) jenkins-bot: Fix a couple base test case things [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381499 (owner: Ejegg) [18:30:43] (CR) Mepps: [C: 1] "This looks good, but are there any tests needed for DonationData or DonationApi? If not, I'm happy to +2." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/383283 (https://phabricator.wikimedia.org/T177663) (owner: Ejegg) [18:40:09] XenoRyet are you comfortable reviewing: https://gerrit.wikimedia.org/r/#/c/384065/ [18:41:17] I could take a look, sure. Haven't had my head in that space in a while, but I'll get into it. [18:42:17] Gonna try to give the baby a bottle, then I'll look into it. [18:53:05] :baby [18:53:12] oh man i was hoping there was an emoji there [18:53:18] 👶 [18:53:20] there [18:53:29] 🍼 [18:58:46] Yea, she's not doing so well on the bottle training, so we're trying to have me give her more bottles durring the day to work on it. [18:58:49] fr-tech pcoombe: impression rates for all four BE tests seem fine when queried on Druid: https://phabricator.wikimedia.org/T177328#3684001 [19:01:24] (CR) XenoRyet: [C: 2] "Looks good" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/384065 (owner: Mepps) [19:05:18] (Merged) jenkins-bot: Fix test + bug for orphan slayer [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/384065 (owner: Mepps) [19:09:24] Fundraising-Backlog, fundraising-tech-ops: fundraising database replication lag master thread - https://phabricator.wikimedia.org/T173472#3684015 (Jgreen) [19:09:26] Fundraising-Backlog, fundraising-tech-ops: start testing mariadb 10.1.23 for fundraising - https://phabricator.wikimedia.org/T176489#3684013 (Jgreen) Open>Resolved >>! In T176489#3634155, @Jgreen wrote: > frdb1003 is up and running stretch with stock mariadb 10.1.23, we'll see how it does So far... [19:09:58] thanks XenoRyet! [19:10:13] No worries [19:10:45] reviewing documentation ahead of Jack starting: have you noticed the onboarding documentation doesn't exist https://www.mediawiki.org/wiki/Fundraising_tech#Documents? [19:13:14] Yea, didn't we have one of those around? [19:13:16] What happened to that? [19:13:22] fundraising-tech-ops: fix aiderator bug around initializing a new host - https://phabricator.wikimedia.org/T170342#3684027 (Jgreen) p:Triage>Normal [19:13:35] fundraising-tech-ops: fix fundraising_code_update process-control git revision reporting bug - https://phabricator.wikimedia.org/T170341#3684028 (Jgreen) p:Triage>Normal [19:13:56] fundraising-tech-ops: blockers for migrating fundraising to stretch - https://phabricator.wikimedia.org/T176655#3684047 (Jgreen) p:Triage>Normal [19:14:40] fundraising-tech-ops, Operations, netops: bonded/redundant network connections for fundraising hosts - https://phabricator.wikimedia.org/T171962#3684051 (Jgreen) p:Triage>Normal [19:14:59] fundraising-tech-ops, Operations, netops, ops-codfw: connect second ethernet interface for fundraising codfw hosts - https://phabricator.wikimedia.org/T176175#3684053 (Jgreen) p:Triage>Normal [19:15:21] fundraising-tech-ops: frdb1002 swap use spike associated with mysql dump run - https://phabricator.wikimedia.org/T177982#3684054 (Jgreen) p:Triage>Normal [19:16:59] not sure--I only saw this page three months into working here! [19:20:12] My onboarding email pointed me at this: https://office.wikimedia.org/wiki/Guide_for_new_engineering_staff [19:20:28] but it's not fr-tech specific.