[00:00:10] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [00:00:30] how curious [00:00:40] i'm going to try restarting the slave process [00:05:10] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1283 [00:06:20] I'm going to step away from chat but I'll be near my phone and back on email by 6ish [00:10:10] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1583 [00:11:25] That escalated quickly. [00:15:10] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 4230485 Threads: 7 Questions: 481232828 Slow queries: 413504 Opens: 2769 Flush tables: 1 Open tables: 1273 Queries per second avg: 113.753 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [00:23:36] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18, Patch-For-Review: Merge on addresses for MG - https://phabricator.wikimedia.org/T181088#3779205 (Eileenmcnaughton) a:Eileenmcnaughton [03:42:23] Going by number of donations rather than total dollars, today was the second biggest day ever [03:42:58] after Dec 3, 2014 [14:24:19] good morning fr-tech! [14:25:38] hi mepps! [14:25:49] hi ejegg|sorta [14:27:09] how goes it? anything you need help on? [14:27:31] i'll be fully available in 15 min... just reading emails from phone now [14:28:45] want to see if you can figure out how we're trying to insert emails or addresses after contact inserts get rolled back? [14:30:28] mepps: ejegg|sorta fr-tech morrrrning [14:33:51] AndyRussG: I noticed some weird behaviour in banner logs starting at noon, although it seems to have recovered now. https://goo.gl/UGP11n [14:34:00] hi AndyRussG [14:34:17] afaik we didn't see any drop in donations [14:38:55] hmm sure ejegg [14:39:46] thanks! eileen and i have been pretty puzzled [15:00:58] hmm okay ejegg i have an idea [15:01:27] or at least the start of an idea [15:01:50] did we deploy the thing to avoid duplicates in emails? [15:03:21] oh wait, hang on [15:03:31] but the interesting thing is that the contacts are in the db [15:07:15] just under different contact ids [15:07:19] not sure if you already got there [15:11:28] pcoombe: arrrg tht does look weird! cps asked about some issue on one of the stats machines, maybe related... checking.. [15:11:31] thx!! [15:14:04] mepps I think they're only inserted b/c I re-queued the damaged messages [15:14:09] let's see... [15:15:02] ohh, but... are some of them inserted twice, once with no address or donation? [15:34:20] pcoombe: yeah that sure looks like an issue with how the data is getting to us. I just asked on the analytics channel [15:34:57] interesting that it's uniform across all campaigns, not just FR [15:35:15] pcoombe: how do things look when queried from the sampled db? [15:35:22] Thanks AndyRussG. The data in pgehres seems to be fine [15:36:51] pcoombe: fantastic, thanks... well that makes it seem pretty sure it's an analytics infrastructure problem [15:37:05] I wonder if it's only in Druid, or if maybe the Hive data is corrupted [15:38:40] mepps ok, i've got another thought. what if the retry logic is somehow ending up returning the wrong id [15:41:38] pcoombe: just spoke to analytics folks, it was an issue rebooting one of the Druid machines (the store that Pivot queries). The base data in Hive is fine, fortunately, and Druid/Pivot will be corrected/backfilled later.. [15:42:21] hmm that would make sense ejegg [15:43:01] I was wondering where the missing contact id came from [15:43:07] gonna ask cps for more details on the issue they reported, analytics didn't think it was related [15:43:30] mepps ohh, so civicrm does some automatic rollbacks [15:43:56] so if the contact got inserted right, but there was an error afterwards, it might have rolled something back [15:44:51] It's just a mystery where it would roll something back and not cause an exception that would stop us in our tracks [15:48:24] Ah so civi created the contact but then rolled it back? Would we get any indication of that? [15:48:40] usually that happens when civi's throwing an exception [15:48:44] so we notice [15:49:05] trying out some retry scenarios locally [15:51:48] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Stat1005 BH data issue? - https://phabricator.wikimedia.org/T181811#3803125 (DStrine) [15:54:45] AndyRussG: it the conversation here related to the email "Stat1005 BH data issue?" [15:54:52] it=is [15:55:14] I made a task: https://phabricator.wikimedia.org/T181811 [15:58:34] mepps bizarre [15:59:17] when I lock up the contact table so it retries, it's somehow coming back from the retry with contact ID still null [16:00:07] Do you want to get on a call in 15? [16:01:30] AndyRussG: ? [16:02:04] dstrine: hi! [16:02:21] dstrine: I thought it might be, then asked on analytics, but no [16:03:08] I mean, two issues, one, noticed by pcoombe (https://goo.gl/WfKTKG) and explained by analytics as an issue rebooting the machine [16:03:24] the other is the one mentioned by cps [16:03:24] ok so we don't know what's causing the CPS reported outage? Also pcoombe is seeing a different issue? [16:03:32] yes and yes [16:03:58] would you mind making a task for pcoombe 's report? [16:04:00] and the one reported by pcoombe is explained, issue in analytics cluster, data will be backfilled there [16:04:09] oh ok [16:04:14] dstrine: there's nothing we need to do about that one [16:04:21] so it's his issue is going to be fixed? [16:04:22] ok [16:04:23] the backfill is basically automagic in that case [16:04:27] cool [16:04:29] yeah [16:04:53] basically there's a job that runs daily that rebuilds the data there anyway [16:05:08] Just about to ask cps for more details about the issue they reported. analytics didn't have any ideas about what that could be [16:06:06] ok cool I think getting clarity in the CPS report is pretty high priority [16:06:27] had you been looking at it already this morning? [16:06:55] ejegg see question above where I didn’t tag you :), want to get on a call in 10? [16:08:00] dstrine: all I did was ask on analytics about it, nothing else so far. We need more details from them, i.e., what error did they get, from what machine/process, or what exactly happened [16:08:11] (more details from cps I mean) [16:08:44] mepps oh sorry, sure! [16:08:48] ok they are around (closer to your time zone) let me know how it goes [16:10:15] dstrine: yea just sending an e-mail [16:12:27] AndyRussG: Maxy just replied. even though it's resolved, I'd like to know the cause. [16:13:06] man I can't type [16:13:09] Ah heh I just sent the e-mail [16:13:49] dstrine: yes, for that we still need more details then [16:15:30] cool ejegg i'm in the chat [16:25:42] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Civi access for Fundraising Contractor - https://phabricator.wikimedia.org/T181740#3803274 (DStrine) [16:26:29] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Stat1005 BH data issue? - https://phabricator.wikimedia.org/T181811#3803292 (DStrine) [16:26:44] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Stat1005 BH data issue? - https://phabricator.wikimedia.org/T181811#3803125 (DStrine) a:AndyRussG [17:57:42] cwd & Jeff_Green my laptop is here! [17:58:04] very nice [17:58:26] I'm just gonna scan over the guide you already linked me to [17:58:26] i just put debian on the x1 last night [17:58:33] about encrypting the drive [17:58:42] ah, what os are you planning to run? [17:59:01] I'm thinking ubuntu [17:59:11] if that works [17:59:53] woo! [17:59:59] hey all trying to do some triage. ejegg what do you think about this? https://phabricator.wikimedia.org/T181768 [18:00:04] am I cool to just run through the full disk encryption guide myself? [18:00:05] priority? Timing? [18:00:19] and wipe the current windows install? [18:00:27] or should I preserve/partition [18:05:30] jgleeson: repartition, wipe out windows! [18:05:39] I left the EFI partition on mine [18:05:48] AndyRussG: any advice for jgleeson ^^^ ? [18:06:08] dstrine: let's do that in January [18:06:12] probably tiny [18:06:14] thanks ejegg ! [18:06:32] we're not officially using Ingenico for Chile, but it's possible to get to that form [18:06:42] ok who about this one : https://phabricator.wikimedia.org/T181748 [18:06:46] jgleeson: I cleaned all that Windows out of there [18:06:50] who - how :( [18:06:59] AndyRussG: did you leave the EFI partition? [18:07:13] Hmm I don't remember... probably did [18:07:51] On the bare metal I created the boot partition and then the encrypted one for everything else [18:08:12] then partitioned that for home, root, var, etc [18:08:35] though I think a simpler option, probably just as good, is to use the whole (encrypted) partition for everything (other than boot) [18:09:23] here's the testing distribution installer: https://www.debian.org/devel/debian-installer/ [18:09:40] dstrine: that'll be a nice thing to do soon-ish [18:09:59] jgleeson: ubuntu is fine [18:10:18] thanks ejegg AndyRussG cwd [18:10:22] there is a lot of debian at wmf [18:10:23] ejegg: I put it in sprint +1 [18:10:31] I'll investigate that [18:10:38] most of our servers are debian and several debian developers work here [18:10:59] ejegg: this one seems a little higher priority https://phabricator.wikimedia.org/T181735 [18:11:08] but you are free to run whatever you want [18:11:08] at least sprint +1 [18:11:12] except windows [18:11:27] yeah windows it going [18:11:39] it blue screened in my first handouts call [18:11:48] right after me complimenting it [18:12:01] lol [18:12:03] although doesn't new windows come with some sort of ubuntu environment? [18:12:03] jgleeson: only thing is that the latest ubuntu installer might not have the wifi yet [18:12:18] i guess you could theoretically use windows in that case [18:12:20] I have yet to get a blue screen on 10. that's pretty impressive [18:12:26] so if you go that route just copy those firmware files over to usb [18:12:34] iwlwifi [18:13:28] here's a tip for installs: https://www.amazon.com/Edimax-EW-7811Un-150Mbps-Raspberry-Supports/dp/B003MTTJOY?psc=1&SubscriptionId=AKIAILSHYYTFIVPWUY6Q&tag=duckduckgo-ffab-20&linkCode=xm2&camp=2025&creative=165953&creativeASIN=B003MTTJOY [18:13:45] that thing has a driver that's been present in every installer i've tried [18:14:23] hey ejegg how about this one? https://phabricator.wikimedia.org/T181692 Sprint +1? [18:15:39] dstrine sometime in the next month we should get those parent subscr_ids inserted [18:15:52] ok [18:15:54] ejegg does the finger print scanner work with debian? [18:16:09] jgleeson: oh hey, I haven't tried [18:16:27] ejegg: mepps would this be worth looking into if someone is looking at dash stuff? [18:16:28] https://phabricator.wikimedia.org/T181590 [18:17:28] it'd be a nice bug to tackle for a newbie to dash [18:17:49] but maggie's request for # url routing would be more helpful / visible [18:18:31] ok I'll move T181590 to sprint +1 [18:18:31] T181590: dash widgets making extraneous requests - https://phabricator.wikimedia.org/T181590 [18:19:12] analysis column is clean. I like going into the weekend knowing where we are with open topics [18:20:47] jgleeson: good question [18:20:48] ejegg dstrine then should i find and move the hash routing into this sprint? [18:20:53] fprint is available in apt [18:21:02] arch wiki says it works out of the box with that [18:21:25] mepps sounds good to me! [18:21:37] of course it has intel me so you might think twice about exposing vitals [18:22:12] mepps: yeah cool and thanks [18:24:18] there is actually an option in the bios to "disable AMT" [18:24:23] i'm sure it's BS [18:25:26] i'm not seeing one ejegg, i'll create it unless you know where it is [18:25:56] hmm ok [18:26:25] thanks [18:26:31] mepps ooh it got a good number: T120000 [18:26:31] T120000: Use URL fragments to indicate which board you're on - https://phabricator.wikimedia.org/T120000 [18:26:44] it's like a super-advanced killbot [18:28:41] mepps if you're adding a new js library it just has to get listed in the requirejs config: https://github.com/wikimedia/wikimedia-fundraising-dash/blob/master/src/app/require.config.js [18:29:09] Fundraising Dash, Fundraising Sprint Winter Wanderland, Fundraising-Backlog: Use URL fragments to indicate which board you're on - https://phabricator.wikimedia.org/T120000#3803645 (mepps) [18:37:14] ejegg just put together the killbot joke, hahah [18:46:24] (PS1) Ejegg: Debugging for mystery FK errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394629 [18:49:08] (CR) jerkins-bot: [V: -1] Debugging for mystery FK errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394629 (owner: Ejegg) [19:02:47] (PS1) Ejegg: Update gateway list [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394636 [19:04:00] ejegg: are you running buster on your x1? [19:04:32] i am not having any luck with virtualbox from apt and am wondering if you noticed the same [19:09:43] (PS2) Ejegg: Debugging for mystery FK errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394629 [19:11:30] (CR) jerkins-bot: [V: -1] Debugging for mystery FK errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394629 (owner: Ejegg) [19:18:12] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Automatically uncheck the "Enabled" checkbox on expired campaigns - https://phabricator.wikimedia.org/T181839#3803968 (DStrine) [19:30:31] ejegg I need to pop out for an hour but when I get back I'll see if you're free and we can catch up on a the last minute additions to the stats stuff and just confirm everyone is okay before I tag it. The final changes are pushed with a basic readme https://github.com/jackgleeson/stats-collector [19:30:41] sure! [20:36:41] Fundraising Dash, Fundraising Sprint Winter Wanderland, Fundraising-Backlog: Use URL fragments to indicate which board you're on - https://phabricator.wikimedia.org/T120000#3804346 (mepps) a:mepps [20:37:14] (PS3) Ejegg: Debugging for mystery FK errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/394629 [20:49:07] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Civi access for Fundraising Contractor - https://phabricator.wikimedia.org/T181740#3804416 (Eileenmcnaughton) Hi, We need to get a C-level approval on these - e.g an email pasted into t... [21:33:37] (PS1) Mepps: WIP Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) [21:34:29] (CR) jerkins-bot: [V: -1] WIP Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) (owner: Mepps) [21:35:37] (PS2) Mepps: WIP Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) [21:37:54] (CR) jerkins-bot: [V: -1] WIP Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) (owner: Mepps)