[01:27:08] Fundraising Sprint Karma chameleons hide amongst us, Fundraising-Backlog, Fr-Ingenico-integration_2017-18, Unplanned-Sprint-Work: Update Ingenico WX audit parser to differentiate txns from new API - https://phabricator.wikimedia.org/T195337#4223855 (Ejegg) [01:38:16] (PS1) Eileen: Re-save any contacts with a middle name. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/434622 (https://phabricator.wikimedia.org/T195311) [02:00:17] (PS2) Eileen: Re-save any contacts with a middle name. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/434622 (https://phabricator.wikimedia.org/T195311) [02:01:10] Fundraising Sprint Junebugs prefer July, Fundraising Sprint Karma chameleons hide amongst us, Fundraising-Backlog, FR-Ingenico, Patch-For-Review: Ingenico audit wobble? iDEAL donations not in Civi May 1-3 - https://phabricator.wikimedia.org/T194296#4223901 (Ejegg) OK, that fix seemed to do th... [14:43:47] fr-tech so we need to do this pretty soon: T195337 [14:43:48] T195337: Update Ingenico WX audit parser to differentiate txns from new API - https://phabricator.wikimedia.org/T195337 [14:44:05] but I'm having a hard time finding a good way to differentiate [14:44:55] Best I can see is that the MerchantReference second part is generally under 100 in the 'ingenico' txns, and some much bigger number in the 'globalcollect' ones [14:45:07] but that's not guaranteed :( [14:45:09] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[intel-microcode] [14:45:49] I was really hoping that 'PlatformIdNo' would be different for Connect vs WebCollect, but no dice [14:50:09] PROBLEM - check_puppetrun on rigel is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Package[intel-microcode] [14:50:10] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[intel-microcode] [14:50:10] PROBLEM - check_puppetrun on heka is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[intel-microcode] [14:55:10] PROBLEM - check_puppetrun on rigel is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 26 seconds ago with 1 failures. Failed resources (up to 3 shown): Package[intel-microcode] [14:56:33] fundraising-tech-ops: payments syslog should send ingenico_gateway lines to their own file - https://phabricator.wikimedia.org/T195381#4225601 (Ejegg) [15:00:09] PROBLEM - check_puppetrun on rigel is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[intel-microcode] [15:02:44] Jeff_Green: hope you're feeling better today! [15:03:35] oh yeah, thanks. it was just one of those underslept days where staring at the computer harder wasn't working [15:20:09] RECOVERY - check_puppetrun on rigel is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [15:25:08] RECOVERY - check_puppetrun on payments1004 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [15:30:08] RECOVERY - check_puppetrun on heka is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [16:15:54] Fundraising Sprint Karma chameleons hide amongst us, Fundraising-Backlog: make a mini test plan and switch over to new thank you page - https://phabricator.wikimedia.org/T195329#4223354 (Ejegg) As far as I know, it has to do two things: - Set hide cookies for the main project domains - Localize and not l... [16:38:48] Fundraising Sprint Junebugs prefer July, Fundraising Sprint Karma chameleons hide amongst us, Fundraising-Backlog, FR-Ingenico, Patch-For-Review: Ingenico audit wobble? iDEAL donations not in Civi May 1-3 - https://phabricator.wikimedia.org/T194296#4226069 (MBeat33) Awesome news, thank you @E... [16:59:36] (PS3) Ejegg: Re-save any contacts with a middle name. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/434622 (https://phabricator.wikimedia.org/T195311) (owner: Eileen) [17:02:30] (CR) Ejegg: [C: 2] Re-save any contacts with a middle name. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/434622 (https://phabricator.wikimedia.org/T195311) (owner: Eileen) [17:03:37] all I have to do is type "karma" and our sprint comes up [17:08:00] (Merged) jenkins-bot: Re-save any contacts with a middle name. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/434622 (https://phabricator.wikimedia.org/T195311) (owner: Eileen) [17:20:42] fr-tech any news for scrum of scrums? [19:25:58] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, and 2 others: Create job to deliver the eventlogg... - https://phabricator.wikimedia.org/T189820#4226494 [19:28:22] as a precaution before deploying the CN EL JS fix, just turning on EL impressions for the banners used by browser tests in the beta cluster... then gonna run the selenium browser test Jenkins job [19:29:03] dstrine: I prefer "hide" [19:29:12] AndyRussG: as it happens we just got the server side portion of that working [19:29:20] cwd: woohoo! [19:29:53] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, and 2 others: Create job to deliver the eventlogg... - https://phabricator.wikimedia.org/T189820#4226496 [19:30:09] cwd: quick question: as far as you can tell, will the new files be any different at all? [19:30:24] looking [19:30:55] I was trying to find the code that makes those (in the legacy setup) but I got lost because my lack of understanding of puppet [19:31:18] AndyRussG: well they are json [19:31:33] although i thought i changed that [19:31:49] quick pointer to where in the puppet repo I should look to see the actual code that creates the files? [19:32:18] I guess it's a Kafakatee config thingy, no? [19:33:05] AndyRussG: the config files? [19:33:13] kafkatee itself creates the text files... [19:34:19] yea [19:34:34] So we just set a kafkatee config to tell it how to do that, right? [19:34:49] yep [19:34:58] modules/kafkatee/templates/k* [19:38:41] cwd: cool gotcha, thx! [19:38:55] What are kafkatee vs kafkatee_old about? [19:40:09] also where do we get values for the vars that are embedded in the erb? [19:40:16] Sorry to bother u with such basic puppet questions [19:45:02] (off to kid pickup in a sec, I'll get scrollback tho... thx!!!!) [19:49:38] AndyRussG|a-whey: ah sorry i was making coffee...most of that stuff is in the hieradata directory [19:49:56] that's a recent adoption, yaml store of variables [19:50:21] kafkatee_old is what's running on americium, the new one has updated stuff on alnitak [19:50:37] we will reimage americium as strech when we can and _old goes away [19:53:31] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, and 2 others: apply https://gerrit.wikimedia.org/... - https://phabricator.wikimedia.org/T195408#4226610 [19:56:42] the confusing part will be the manifests are an intermediary, potentially, for the hieradata and the templates [19:56:57] where things could change name etc [19:57:03] so it's a lot of grepping [19:57:27] even the basic puppet questions have extremely non-obvious answers [20:17:15] (PS2) Ejegg: No more magic strings for WmfException [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/431695 [20:21:27] (CR) jerkins-bot: [V: -1] No more magic strings for WmfException [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/431695 (owner: Ejegg) [20:38:10] (PS3) Ejegg: No more magic strings for WmfException [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/431695 [20:40:46] fundraising-tech-ops: EPIC: migrate fundraising hosts to Debian Stretch - https://phabricator.wikimedia.org/T185013#4226797 (Jgreen) - ------------------------------------------------------------------------- Debian Security Advisory DSA-4205-1 security@debian.org https://www.debian.org/sec... [20:41:46] fundraising-tech-ops: reimage americium with Jessie - https://phabricator.wikimedia.org/T165393#4226799 (Jgreen) [20:41:48] fundraising-tech-ops: reimage bismuth to Jessie - https://phabricator.wikimedia.org/T165392#4226800 (Jgreen) [20:41:50] fundraising-tech-ops: EPIC: migrate fundraising hosts to Debian Stretch - https://phabricator.wikimedia.org/T185013#4226798 (Jgreen) [20:42:47] fundraising-tech-ops: upgrade all frack servers to debian/jessie - https://phabricator.wikimedia.org/T146479#4226802 (Jgreen) [20:42:49] fundraising-tech-ops: EPIC: migrate fundraising hosts to Debian Stretch - https://phabricator.wikimedia.org/T185013#3903328 (Jgreen) [20:43:25] fundraising-tech-ops: reimage bismuth to Jessie - https://phabricator.wikimedia.org/T165392#3264884 (Jgreen) [20:43:47] fundraising-tech-ops: reimage americium to Debian Stretch - https://phabricator.wikimedia.org/T165393#4226805 (Jgreen) [20:44:22] fundraising-tech-ops: reimage bismuth to Debian Stretch - https://phabricator.wikimedia.org/T165392#4226807 (Jgreen) [20:44:51] fundraising-tech-ops: EPIC: migrate fundraising hosts to Debian Stretch - https://phabricator.wikimedia.org/T185013#4226810 (Jgreen) [20:44:53] fundraising-tech-ops: upgrade all frack servers to debian/jessie - https://phabricator.wikimedia.org/T146479#2662370 (Jgreen) Open>Resolved [20:45:15] fundraising-tech-ops: reimage tellurium to jessie - https://phabricator.wikimedia.org/T165391#4226812 (Jgreen) [20:45:17] fundraising-tech-ops: reimage bismuth to Debian Stretch - https://phabricator.wikimedia.org/T165392#3264884 (Jgreen) [20:45:27] fundraising-tech-ops: reimage bismuth to Debian Stretch - https://phabricator.wikimedia.org/T165392#3264884 (Jgreen) [20:45:29] fundraising-tech-ops: reimage americium to Debian Stretch - https://phabricator.wikimedia.org/T165393#3264897 (Jgreen) [20:46:46] fundraising-tech-ops, Operations, ops-eqiad: rack frdata1001 - https://phabricator.wikimedia.org/T187364#4226818 (Jgreen) a:Jgreen>None [20:48:09] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: Rack/setup frmon1001 - https://phabricator.wikimedia.org/T186073#4226819 (Jgreen) a:Jgreen>None [20:50:42] Fundraising-Backlog, fundraising-tech-ops: Alert fatigue: "frdev1001/check_procs is CRITICAL" - https://phabricator.wikimedia.org/T191798#4116823 (Jgreen) Simple/reasonable fix would be to adjust the alert threshold to the high water mark of those events. [20:59:05] fr-tech: got this CRM patch to review that's not very tricky, but touches a lot of files: https://gerrit.wikimedia.org/r/431695 [21:02:27] fundraising-tech-ops: payments syslog should send ingenico_gateway lines to their own file - https://phabricator.wikimedia.org/T195381#4226836 (Jgreen) a:Ejegg I adjusted the rsyslog config and logs should be handled as described. I didn't see any ingenico logs coming in at the moment, once you can confi... [21:10:25] fundraising-tech-ops: unattended-upgrades on frack - https://phabricator.wikimedia.org/T191907#4226870 (cwdent) Open>Resolved Works errrywhere [21:13:03] fundraising-tech-ops: adopt apt.wikimedia.org on frack - https://phabricator.wikimedia.org/T188668#4226872 (cwdent) Open>declined We have continued to find it manageable to just import the things we need so not gonna do this ATM [22:17:42] /nick AndyRussG|ish [22:18:06] cwd: okok thanks! :) [22:35:14] (PS4) Ejegg: No more magic strings for WmfException [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/431695 [22:39:40] fundraising-tech-ops: payments syslog should send ingenico_gateway lines to their own file - https://phabricator.wikimedia.org/T195381#4227136 (Ejegg) Open>Resolved Thanks @Jgreen. There's no traffic directed at the gateway just yet, so I just hit it with a few test requests. Seems to be working! [22:40:16] AndyRussG|ish: hi! Just saw some of your questions in the backlog about the new format for request data [22:40:58] it will be fairly different - the old job expected to read apache log files, so when we started getting data from kafka [22:41:26] Jeff_Green wrote a thing to un-parse the kafka data and make it look like apache web logs again [22:41:46] I figured now would be a good time to stop that silliness and just read the json in as it comes off of the kafka topic [22:43:30] XenoRyet: any clues where the spurious entries in that list were coming from? [22:43:35] want to talk it out live? [22:49:52] ejegg: not sure if you saw but that collector is working now [22:50:03] eventlogging_CentralNoticeImpression [22:50:18] awesome! [22:50:22] it's all coming together [22:50:27] :) [22:50:34] so we just need to stuff it in mysql? [22:58:45] cwd there's a tiny amount of filtering [23:17:27] cwd: yea there's a python script that I was hoping to barely touch [23:18:13] AndyRussG|ish: it'll need a few changes, but hopefully mostly deletions [23:18:14] ejegg: are you sure now's the time? also how does that fit in with what cwd got working? [23:18:52] AndyRussG|ish: well, cwd got the raw json from EL dumped to files [23:19:15] he didn't recreate the thing Jeff_Green made to turn it back into pretend apache logs [23:19:43] ejegg: ah okok [23:20:18] is that the most recent change in the fr puppet repo? [23:21:56] hmm there are quite a few recentish changes [23:22:11] ejegg: cwd: you wanted to do a video call about it? [23:22:53] I can do so but would need about 10-15 min advance warning...... ;p [23:23:41] ejegg: so the idea would be to adapt the Python script to read the kafka data? [23:24:05] AndyRussG|ish: yeah, i can do a call in 15 min! [23:24:15] Counterargument: bigger changes are coming to analytics pretty soon, and it might be a better idea to do refactoring after that [23:24:46] AndyRussG|ish: we're already refactoring, and would either have to write a new piece of code to make fake apache logs [23:24:58] ah hmmm I see [23:25:11] ok so back in 15 min! :) [23:25:19] thx! [23:25:19] k, cool! [23:28:49] ejegg: if you want to explain this a little I can review - https://gerrit.wikimedia.org/r/#/c/431695/4/sites/all/modules/globalcollect_audit/globalcollect_audit.module [23:29:57] looks like you switched to class constants https://gerrit.wikimedia.org/r/#/c/431695/4/sites/all/modules/wmf_common/WmfException.php [23:30:39] eileen: yes! [23:31:04] Basically, doing it like you assumed it was supposed to work in one of your recent patches [23:31:15] and avoiding the possibility of misspelling the things [23:31:23] :-) [23:31:39] eileen: oops, that globalcollect_audit thing is wrong though [23:31:56] it just happened to share the strings [23:32:26] I'm actually deleting that whole obsolete module in https://gerrit.wikimedia.org/r/434597 [23:34:20] cwd: r u also around? if you'd like to join, is around 40 past the hour good for you? or is another time better? [23:35:31] eileen I'll undo that file edit [23:37:06] ejegg: OK - & I'll google the way you used ReflectionClass :-) [23:37:43] (PS5) Ejegg: No more magic strings for WmfException [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/431695 [23:38:15] heh, just a way to get those magic strings back when we wanted them for email subjects [23:38:39] ejegg: it looks like the parameter is a string not an object though? http://php.net/manual/en/class.reflectionclass.php [23:38:43] AndyRussG|ish: sorry, give me 5 more minutes? Laundry just stopped spinning [23:43:38] ejegg: sure! [23:46:20] ok AndyRussG|ish, popping into the usual hangout [23:47:23] okok [23:49:13] ejegg: hmmm having some chromium issues one sec