[00:09:48] (PS1) Ejegg: Output simple list of running jobs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/381905 [00:18:13] (PS3) Ejegg: Process-control code cleanup [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/381278 [00:18:15] (PS2) Ejegg: Output simple list of running jobs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/381905 [00:39:26] (PS1) Eileen: Fix UI batch merge mishandling context paramter. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/381910 (https://phabricator.wikimedia.org/T176256) [00:43:50] (PS1) Eileen: CRM-21202, set retrieve = true so that its not reinitialized again and just want the DataTable instance to be returned [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/381912 (https://phabricator.wikimedia.org/T176256) [01:01:59] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: New payments-wiki errors: We don't have an order status after doing a GET_ORDERSTATUS - https://phabricator.wikimedia.org/T176355#3652747 (Ejegg) Well, they're all related to 20001000 NULL VALUE NO... [01:12:31] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: New payments-wiki errors: We don't have an order status after doing a GET_ORDERSTATUS - https://phabricator.wikimedia.org/T176355#3652752 (Ejegg) Aha, we were previously looking at the raw STATUSID... [01:39:42] (PS1) Ejegg: Fix missing order status [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/381918 (https://phabricator.wikimedia.org/T176355) [01:52:20] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 5 others: Are we losing transactions witih repeated ct_id? - https://phabricator.wikimedia.org/T171349#3652766 (Ejegg) Open>Resolved Fourth time's a... [02:44:22] (Abandoned) Eileen: Fix UI batch merge mishandling context paramter. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/381910 (https://phabricator.wikimedia.org/T176256) (owner: Eileen) [02:44:31] (PS2) Eileen: CRM-21202, set retrieve = true so that its not reinitialized again and just want the DataTable instance to be returned [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/381912 (https://phabricator.wikimedia.org/T176256) [02:44:33] (PS1) Eileen: Fix UI batch merge mishandling context paramter. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/381920 (https://phabricator.wikimedia.org/T176256) [02:44:35] (PS1) Eileen: CRM--21248 Fix merge screen conflict listing [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/381921 (https://phabricator.wikimedia.org/T176256) [02:49:36] (CR) Eileen: "The whole criteria in url is not fully upstreamed yet - hence confusion" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/381920 (https://phabricator.wikimedia.org/T176256) (owner: Eileen) [03:26:17] (PS1) Eileen: CRM-21224 get LIMIT out of the where string. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/381922 [03:52:09] (PS3) Eileen: The big group clean up. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/380667 (https://phabricator.wikimedia.org/T174407) [03:52:11] (PS1) Eileen: Fix ImportMessageTest to actually test contact. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381923 (https://phabricator.wikimedia.org/T175744) [03:55:45] (CR) jerkins-bot: [V: -1] Fix ImportMessageTest to actually test contact. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381923 (https://phabricator.wikimedia.org/T175744) (owner: Eileen) [14:11:02] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, fundraising-tech-ops, Unplanned-Sprint-Work: civicrm mail settings vs outcome snafu - https://phabricator.wikimedia.org/T177244#3653613 (Jgreen) @Ejegg I parsed civi mail logs going back to 7/12 when this changed and there appear to... [15:33:37] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, fundraising-tech-ops, Unplanned-Sprint-Work: civicrm mail settings vs outcome snafu - https://phabricator.wikimedia.org/T177244#3653866 (Ejegg) Thanks @Jgreen, I'll turn those into a list of donations. @MBeat, these are donors who d... [15:35:05] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, fundraising-tech-ops, Unplanned-Sprint-Work: civicrm mail settings vs outcome snafu - https://phabricator.wikimedia.org/T177244#3653870 (MBeat33) That looks perfect, @Ejegg thank you for the followup. [15:40:14] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 57 seconds ago with 1 failures. Failed resources (up to 3 shown): File[/var/spool/prometheus/syslog_collector.prom] [15:45:07] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/var/spool/prometheus/syslog_collector.prom] [15:45:08] PROBLEM - check_puppetrun on tellurium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 59 seconds ago with 1 failures. Failed resources (up to 3 shown): File[/var/spool/prometheus/syslog_collector.prom] [15:45:08] PROBLEM - check_puppetrun on alnitak is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/var/spool/prometheus/syslog_collector.prom] [15:50:08] PROBLEM - check_puppetrun on tellurium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/var/spool/prometheus/syslog_collector.prom] [15:50:09] RECOVERY - check_puppetrun on americium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:50:09] PROBLEM - check_puppetrun on alnitak is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/var/spool/prometheus/syslog_collector.prom] [15:50:10] PROBLEM - check_puppetrun on heka is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/var/spool/prometheus/syslog_collector.prom] [15:50:39] grr. known, fixing ^^^ [15:55:16] RECOVERY - check_puppetrun on tellurium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [15:55:17] RECOVERY - check_puppetrun on alnitak is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:55:17] PROBLEM - check_puppetrun on heka is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 13 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/var/spool/prometheus/syslog_collector.prom] [16:00:16] RECOVERY - check_puppetrun on heka is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [16:00:16] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): User[prometheus] [16:05:16] RECOVERY - check_puppetrun on frdb2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:05:37] Hey @ejegg @MBeat we are ready to launch France banners. Any objections? [16:05:56] nope, go for it jessicarobell [16:06:07] nope, not from fr-tech! [16:06:51] Great! [16:06:53] TY! [16:09:36] France desktop campaigns are enabled! :) @ejegg @MBeat @Seddon [16:26:35] jessicarobell: I can tell :P Twitter just lit up :P [16:26:54] (CR) XenoRyet: [C: 2] Fix missing order status [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/381918 (https://phabricator.wikimedia.org/T176355) (owner: Ejegg) [16:27:01] hehe! more Jimmy bashing . Poor Jimmy [16:29:15] jessicarobell: You know we are still using #keepitfree for the twitter social media for french? [16:29:48] (Merged) jenkins-bot: Fix missing order status [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/381918 (https://phabricator.wikimedia.org/T176355) (owner: Ejegg) [16:35:38] thanks XenoRyet [16:35:44] I'll get that fix out [16:35:55] No worries, looked good and I know that was one of the higher priority fixes. [16:37:23] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/382018 [16:37:25] Fictional Jimbo Wales: http://www.everytopicintheuniverseexceptchickens.com/ [16:37:26] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/382018 (owner: Ejegg) [16:38:38] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/382019 [16:38:42] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/382019 (owner: Ejegg) [16:38:50] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/382018 (owner: Ejegg) [16:43:58] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/382019 (owner: Ejegg) [16:45:11] PROBLEM - check_puppetrun on alnilam is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): User[prometheus] [16:50:10] RECOVERY - check_puppetrun on alnilam is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [16:51:13] !log updated payments-wiki from c2a3d434d1234c3ae62b6b0359d69f302bca3bd6 to b9859f6c0efd761d0a87aa154337f5ec91153800 [16:51:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:52:24] XenoRyet: looks like there's an opportunity to deploy process-control today [16:52:55] There's one more thing I'd like to get merged if possible, that would let us add job runs to prometheus [16:53:12] https://gerrit.wikimedia.org/r/381905 [16:53:45] Looking [16:53:51] oh, and that's got a tiny bit of code cleanup preceding it: https://gerrit.wikimedia.org/r/381278 [16:54:03] 10-4 [16:55:08] (CR) XenoRyet: [C: 2] Process-control code cleanup [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/381278 (owner: Ejegg) [16:55:36] (Merged) jenkins-bot: Process-control code cleanup [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/381278 (owner: Ejegg) [16:58:25] (CR) XenoRyet: [C: 2] Output simple list of running jobs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/381905 (owner: Ejegg) [16:59:17] (Merged) jenkins-bot: Output simple list of running jobs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/381905 (owner: Ejegg) [17:01:41] Fundraising-Backlog: Minor process-control lock refactor for separation of concerns - https://phabricator.wikimedia.org/T177296#3654049 (Ejegg) [17:01:57] Oh hey, XenoRyet, did you figure out what it'll take to get a new tag for the process-control project? ^^^ [17:02:55] Kinda. I can't just do it, nor can David. Basically one of us would have to write up a kind of proposal about why we need it, then submit it to a group and they decide if it's worth creating or not. [17:03:11] I can start working on it, but apparently they're kind of picky about it. [17:03:30] Trying to keep phab from getting flooded with tags is the goal, I think. [17:04:17] Still, since they're cool with the payment processor related ones, a process-control tag shouldn't be too hard to get. [17:04:37] Dang, I thought David said it all got easier recently, and that was how we got project tags like CiviCRM dedupe and Ingenico reintegration [17:05:03] Maybe he knows something I don't. I'll look again. [17:05:31] Thanks, it would be nice to have a tag for that thing [17:05:38] Yep, agreed. [17:31:04] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, fundraising-tech-ops, Unplanned-Sprint-Work: civicrm mail settings vs outcome snafu - https://phabricator.wikimedia.org/T177244#3654165 (Ejegg) OK, these are all re-sent [19:01:33] Fundraising-Backlog: Ask GlobalCollect to let us cancel status 600 transactions - https://phabricator.wikimedia.org/T114205#3654727 (Ppena) Update: the networks started charging us penalties for trnxs not canceled within 72 hrs. Donor Services won't have the bandwidth to look at these, let alone cancel them... [19:18:37] ejegg|food: turns out the e-check fraud PayPal is looking at is from 2 weeks ago, so I won’t need to update T171349, sorry for the scare [19:18:38] T171349: Are we losing transactions witih repeated ct_id? - https://phabricator.wikimedia.org/T171349 [19:22:46] Fundraising-Backlog: Impressions from German chapter - https://phabricator.wikimedia.org/T177325#3654829 (jrobell) [19:29:37] (PS1) Mepps: T177230 Thank you letter update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382034 [19:33:35] Fundraising-Backlog: Impressions from Big English tests - https://phabricator.wikimedia.org/T177328#3654890 (jrobell) [19:35:53] ejegg i ran make-thank-you but i'm not seeing the office address update ^^ [19:46:49] mepps XenoRyet standup? [19:46:59] Yea, checking one last thing then brt [19:50:19] Fundraising-Backlog: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3654959 (CCogdill_WMF) [19:50:46] hey, if anyone has a moment to look at the above task, I will be forever grateful! :) [19:53:44] Fundraising-Backlog: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3654976 (CCogdill_WMF) [19:56:56] (CR) Zoranzoki21: [C: 1] T177230 Thank you letter update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382034 (owner: Mepps) [20:00:22] ccogdill: we're in standup now, I'll bring it up! [20:00:27] thank you! [20:02:48] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3654992 (XenoRyet) [20:11:16] fundraising-tech-ops, Operations, netops, ops-codfw: vcp port down on fasw-c-codfw - https://phabricator.wikimedia.org/T177332#3655009 (ayounsi) [20:11:32] fundraising-tech-ops, Operations, netops, ops-codfw: vcp port down on fasw-c-codfw - https://phabricator.wikimedia.org/T177332#3655022 (ayounsi) [20:23:18] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3655034 (Ejegg) @Jgreen Could there be something wrong with the log collector? I'm seeing entries where a contact_id shows up in a... [20:23:53] fundraising-tech-ops, Operations, netops, ops-eqiad: Interface error on fasw-c-eqiad:vcp-255/1/0 - https://phabricator.wikimedia.org/T177333#3655036 (ayounsi) [20:52:27] (PS1) Eileen: [wip] Remove duplicate spaces in name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382062 (https://phabricator.wikimedia.org/T175744) [20:56:58] (PS2) Eileen: Remove duplicate spaces in name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382062 (https://phabricator.wikimedia.org/T175744) [21:02:50] (CR) jerkins-bot: [V: -1] Remove duplicate spaces in name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382062 (https://phabricator.wikimedia.org/T175744) (owner: Eileen) [21:03:33] Fundraising-Backlog, Gerrit, Release-Engineering-Team (Kanban): Determine status of fundraising-related branches and tags - https://phabricator.wikimedia.org/T177334#3655092 (demon) [21:16:50] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3655129 (Ejegg) @CCogdill_WMF did you used to just link to https://donate.wikimedia.org/?country=US& Or have you be... [21:31:03] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3655187 (CCogdill_WMF) Sorry, no, here's an actual link from a template: https://donate.wikimedia.org/?utm_campaign=C1718_Email1&u... [22:14:54] (PS3) Eileen: Remove duplicate spaces in name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382062 (https://phabricator.wikimedia.org/T175744) [22:14:56] (PS2) Eileen: Fix ImportMessageTest to actually test contact. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381923 (https://phabricator.wikimedia.org/T175744) [22:22:20] (CR) jerkins-bot: [V: -1] Remove duplicate spaces in name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382062 (https://phabricator.wikimedia.org/T175744) (owner: Eileen) [22:22:23] (CR) jerkins-bot: [V: -1] Fix ImportMessageTest to actually test contact. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381923 (https://phabricator.wikimedia.org/T175744) (owner: Eileen) [22:37:40] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3655428 (Ejegg) @pcoombe and @MeganHernandez_WMF: There seem to be lines missing in the logs we use to feed the pgehres database, w... [22:38:14] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3655431 (Ejegg) The lines are all still in the main Hive web logs, so nothing is permanently lost. [22:38:50] (PS3) Eileen: Fix ImportMessageTest to actually test contact. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381923 (https://phabricator.wikimedia.org/T175744) [22:39:48] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3655433 (spatton) Hey @Ejegg I'm here and that last comment from you catches my eye! So would we be having the same issues as Caitl... [22:41:32] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Unplanned-Sprint-Work: Email click data not tracking properly - https://phabricator.wikimedia.org/T177331#3655452 (Ejegg) @spatton it's definitely possible. These would be the landing page counts in pgehres that are off. Any numbers you'... [22:53:31] (PS4) Eileen: Fix ImportMessageTest to actually test contact. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381923 (https://phabricator.wikimedia.org/T175744) [22:53:33] (PS4) Eileen: Remove duplicate spaces in name fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/382062 (https://phabricator.wikimedia.org/T175744) [23:20:43] PROBLEM - check_swap on civi1001 is CRITICAL: SWAP CRITICAL - 75% free (5711 MB out of 7623 MB) [23:21:00] ohh Jeff_Green what is that? [23:21:08] cwd ? [23:21:23] i think it's ejegg's grep [23:21:33] :-) [23:21:35] yep, that's me [23:21:47] you have upset ichinga [23:22:06] inverse-grepping 25 days of contact ids to see which ones only showed up in referrers [23:22:17] and when it started [23:23:10] ah, just 15 days worth [23:23:32] but... the problem is present from the first day I checked [23:24:01] can you check an arbitrary date? [23:25:27] if so check around 6/20 vs 6/25 [23:25:37] that's the last kafkatee upgrade [23:33:54] sure thing [23:34:17] one day at a time should be less stressful on the server [23:34:36] oh right, mintaka will have that stuff [23:52:46] !log stopped all jobs for process-control upgrade [23:52:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log