[07:05:10] PROBLEM - check_procs on frdev1001 is CRITICAL: PROCS CRITICAL: 1141 processes [07:10:10] RECOVERY - check_procs on frdev1001 is OK: PROCS OK: 280 processes [13:11:16] (PS2) Gopavasanth: Remove 'UnitTestList' hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428059 (https://phabricator.wikimedia.org/T142120) [13:12:19] (CR) Gopavasanth: "> Uploaded patch set 2." (2 comments) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428059 (https://phabricator.wikimedia.org/T142120) (owner: Gopavasanth) [14:30:27] Fundraising-Backlog: Edit bitpay page inside their console - https://phabricator.wikimedia.org/T192790#4149925 (DStrine) [15:09:08] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog: Ingenico audit wobble? March 13 refunds not in Civi - https://phabricator.wikimedia.org/T190098#4150093 (MBeat33) hi @mepps another data point, Ingeni... [15:15:35] (PS1) Ejegg: Fix typo [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/428348 [15:18:22] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Update silverpop export to use new do_not_solicit field - https://phabricator.wikimedia.org/T192517#4141682 (mepps) We got multiple failmails over the weekend... [15:27:58] (CR) Mepps: [C: 2] Fix typo [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/428348 (owner: Ejegg) [15:28:37] (Merged) jenkins-bot: Fix typo [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/428348 (owner: Ejegg) [15:29:26] Fundraising-Backlog, FR-Adyen, FR-Smashpig: Typo in Adyen payment cancel - https://phabricator.wikimedia.org/T192803#4150160 (Ejegg) [15:50:01] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi change log yielding server error 4/23/18 - https://phabricator.wikimedia.org/T192808#4150242 (MBeat33) [15:50:11] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:55:10] RECOVERY - check_puppetrun on frdb2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [15:56:50] (PS1) Ejegg: Specify collation on country join [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/428360 [15:57:08] mepps: does ^^^ fix it for you locally? [16:07:51] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi change log yielding server error 4/23/18 - https://phabricator.wikimedia.org/T192808#4150390 (Ejegg) Looks like it's missing a table name - note that `civicrm`. entity_log_civireport has a space after the '.' ``` INSERT IGNORE INTO civicrm_temp_civ... [16:10:58] hmm, pings still muted? [16:15:14] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:20:14] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:25:14] RECOVERY - check_puppetrun on frdb2001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:32:51] Fundraising-Backlog: Create a test system to check banners using CSP headers on banner changes - https://phabricator.wikimedia.org/T191915#4150522 (AndyRussG) Some thoughts ideas on how do to do this: - Initially we could run a job on labs while we work out what the best setup will be. - It seems we won't ne... [16:34:07] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Browser-Tests, Patch-For-Review, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#4150531 (greg) [16:36:40] ejegg that did not fix it :( [16:36:50] hmm i should see why i'm not seeing them ejegg [16:40:47] ejegg do you want me to work on it since it's not happening for you locally? [16:41:24] oh no wait! [16:41:31] i've got a different error, hang on [16:44:56] i'm having issues with the drupal update [16:50:43] mepps oh... the one to add the new field? [16:51:01] yeah it looks like i'm hitting a fatal error before then [16:51:12] what's the error say? [16:52:53] it just says fatal error triggered :( [16:53:05] but for now i'm skipping htat update--it's just to resave blank contacts [17:00:53] oh weird [17:24:01] phew, more adyen txn stragglers? [17:43:35] if they’re a separate thing i’m happy to split into a new task [18:25:20] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: [Epic] CentralNotice performance - https://phabricator.wikimedia.org/T192828#4150938 (AndyRussG) [18:27:24] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Epic: [Epic] CentralNotice performance - https://phabricator.wikimedia.org/T192828#4150950 (AndyRussG) [18:27:53] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Epic: [Epic] Banner history improvements - https://phabricator.wikimedia.org/T192564#4150951 (AndyRussG) [18:29:12] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Epic: [Epic] CentralNotice performance - https://phabricator.wikimedia.org/T192828#4150938 (AndyRussG) For current discussion on this, see the [[ https://docs.google.com/document/d/1bi8SPRFKzoo57cf-vcPQ5fnfN_RyWlAgGRJl2t6vocA/edit# | CentralNot... [18:31:11] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: CentralNotice: Instrument performance metrics - https://phabricator.wikimedia.org/T192829#4150960 (AndyRussG) [18:31:35] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Epic: [Epic] CentralNotice performance - https://phabricator.wikimedia.org/T192828#4150972 (AndyRussG) [18:31:40] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: CentralNotice: Instrument performance metrics - https://phabricator.wikimedia.org/T192829#4150971 (AndyRussG) [18:40:50] mepps any luck on your separate problem? [18:50:59] (PS3) Umherirrender: Remove 'UnitTestList' hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428059 (https://phabricator.wikimedia.org/T142120) (owner: Gopavasanth) [18:51:03] (PS4) Umherirrender: Remove 'UnitTestList' hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428059 (https://phabricator.wikimedia.org/T142120) (owner: Gopavasanth) [18:54:19] (CR) jerkins-bot: [V: -1] Remove 'UnitTestList' hook [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428059 (https://phabricator.wikimedia.org/T142120) (owner: Gopavasanth) [18:58:00] (CR) Ejegg: Remove 'UnitTestList' hook (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/428059 (https://phabricator.wikimedia.org/T142120) (owner: Gopavasanth) [18:59:15] mepps are you still not getting pings from irccloud? [18:59:39] ejegg no, this time i've been in meetings for the past hour and a half :P [19:00:52] Ah, got it! [19:02:45] ejegg i was running into some issues with my local crm setup before, but since i was getting a new and different error message i wonder if we can assume your fix works? i don't love that logic... [19:04:19] well, does your error happen after that query at least? [19:15:53] nope, good catch--sorry today is my busy meeting day so i've been distracted [19:27:51] Fundraising-Backlog: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression data - https://phabricator.wikimedia.org/T192839#4151180 (AndyRussG) [19:28:51] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, Epic: [Epic] Kafkatee changes - https://phabricator.wikimedia.org/T183978#4151191 (AndyRussG)... [19:30:00] (PS5) Jgleeson: WIP - Added Mapper Transformer behaviour [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/427723 [19:31:25] Fundraising-Backlog: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression data - https://phabricator.wikimedia.org/T192839#4151205 (AndyRussG) It has been suggested that we use [[ https://github.com/dpkp/kafka-python | python-kafka ]] now... Would this as a complement to, o... [19:35:53] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Unplanned-Sprint-Work: Civi change log yielding server error 4/23/18 - https://phabricator.wikimedia.org/T192808#4151234 (DStrine) [19:36:50] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog, FR-Adyen, FR-Smashpig, Unplanned-Sprint-Work: Typo in Adyen payment cancel - https://phabricator.wikimedia.org/T192803#4151238 (DStrine) [19:38:20] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog: [Spike] Plan ingress system to use with new Kafka topic for landing page and impression data - https://phabricator.wikimedia.org/T192839#4151180 (DStrine) [19:41:46] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog, Patch-For-Review: Refund the unintended recurring donations from the tests pre big english - https://phabricator.wikimedia.org/T190854#4151262 (MBe... [19:43:10] Fundraising-Backlog, FR-Smashpig, Patch-For-Review: Queue retry logic shouldn't use original enqueued date - https://phabricator.wikimedia.org/T192692#4151263 (DStrine) [19:45:04] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Banner History: Include some banner performance metrics in log - https://phabricator.wikimedia.org/T192842#4151270 (AndyRussG) [19:45:44] (PS6) Jgleeson: WIP - Added Mapper Transformer behaviour [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/427723 [19:46:13] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Epic: [Epic] CentralNotice performance - https://phabricator.wikimedia.org/T192828#4151284 (AndyRussG) [19:46:16] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Epic: [Epic] Banner history improvements - https://phabricator.wikimedia.org/T192564#4151285 (AndyRussG) [19:46:19] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Banner History: Include some banner performance metrics in log - https://phabricator.wikimedia.org/T192842#4151283 (AndyRussG) [19:49:54] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: Add Banner Guidelines link to banner edi... - https://phabricator.wikimedia.org/T190089#4151292 [19:52:51] Fundraising-Backlog, Fr-CiviCRM-dedupe-FY2017/18: Civi: make results page number sticky for Find & Merge Duplicates - https://phabricator.wikimedia.org/T192161#4151317 (DStrine) [20:01:17] (PS7) Jgleeson: WIP - Added Mapper Transformer behaviour [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/427723 [20:20:30] PROBLEM - check_puppetrun on tellurium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/send_nsca.cfg] [20:25:31] PROBLEM - check_puppetrun on tellurium is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 14 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/send_nsca.cfg] [20:27:47] Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Unplanned-Sprint-Work: Civi change log yielding server error 4/23/18 - https://phabricator.wikimedia.org/T192808#4151487 (Eileenmcnaughton) a:Eileenmcnaughton [20:28:59] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: Add signature images for MG mailings - https://phabricator.wikimedia.org/T189535#4044915 (Eileenmcnaughton... [20:30:11] RECOVERY - check_puppetrun on tellurium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:30:20] PROBLEM - check_puppetrun on payments2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:35:10] PROBLEM - check_puppetrun on bismuth is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:35:20] RECOVERY - check_puppetrun on payments2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [20:37:24] ejegg my local setup is messed up but i can confirm your fix works [20:38:16] mepps great! If you wanna +2 it I'll deploy right away [20:40:10] RECOVERY - check_puppetrun on bismuth is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:45:17] (CR) Mepps: [C: 2] Specify collation on country join [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/428360 (owner: Ejegg) [20:45:29] thanks! [20:45:48] really relocating now, will deploy that from the library [20:45:49] (Merged) jenkins-bot: Specify collation on country join [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/428360 (owner: Ejegg) [20:48:15] catch you all tomorrow fr-tech! have a good one [21:01:18] just a heads up ejegg in case my local instance turns out to foresee all errors on prod, the next error i was getting but of a silverpop field not existing (it was a later query than the other error though) [21:01:41] but my crm instance is so messed up with extensions right now i think it's more likely it's a local issue [21:28:41] mepps ohh, there might indeed be a silverpop table to create [21:28:54] silverpop_countrylangs maybe? [21:35:23] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/428530 [21:36:11] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/428530 (owner: Ejegg) [21:36:36] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/428530 (owner: Ejegg) [21:37:18] (PS1) Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/428531 [21:37:28] (CR) Ejegg: [C: 2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/428531 (owner: Ejegg) [21:37:34] (Merged) jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/428531 (owner: Ejegg) [21:39:53] !log updated SmashPig from 1ebee97a45 to a4de12d415 [21:39:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:40:40] !log updated fundraising python tools from 7c5c7a5f9e to f3ed1d05b8 [21:40:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:42:11] ok, re-running the silverpop export to make sure that didi it [22:00:39] (PS13) Eileen: Drush script to fill totals fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/427319 [22:01:12] (CR) Eileen: "thanks @ejegg - just updated the drush_print to be just a string" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/427319 (owner: Eileen) [22:28:43] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Notify MG when contact crosses $1000 in fiscal year - https://phabricator.wikimedia.org/T192856#4151776 (Eileenmcnaughton) [22:45:59] hmm I got knocked off [22:50:28] (PS1) Eileen: Fix totals field to be search-range searchable [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428538 [22:51:17] ejegg: from the meeting https://gerrit.wikimedia.org/r/#/c/428538/ [22:51:33] eileen: I realize this library is going to close before the outage is over, so I'm heading back to the apt. [22:51:40] back on in 15 min or so! [22:51:41] ejegg: ok cool [22:55:31] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428539 [22:56:01] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428539 (owner: Eileen) [22:57:02] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428539 (owner: Eileen) [23:06:50] (PS2) Eileen: Fix totals field to be search-range searchable [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428538 [23:06:52] (PS14) Eileen: Drush script to fill totals fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/427319 [23:09:48] (CR) jerkins-bot: [V: -1] Drush script to fill totals fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/427319 (owner: Eileen) [23:13:04] !log disabled most (all?) frack alerts [23:13:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:15:58] mepps: are you about? [23:16:31] ejegg is back anyway - ejegg can you cr https://gerrit.wikimedia.org/r/#/c/428538/2/sites/all/modules/wmf_civicrm/wmf_civicrm.install [23:16:39] (change from today's mtg) [23:19:05] for sure! [23:19:51] (CR) Ejegg: [C: 2] Fix totals field to be search-range searchable [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428538 (owner: Eileen) [23:20:17] ejegg: can the listener function w/o db? [23:20:23] cwd yes! [23:20:27] groovy [23:20:28] It just dumps stuff into the queue [23:20:30] i will leave it up then [23:20:32] nice [23:20:33] thanks [23:20:37] i'm just going to take down payments and civi [23:20:49] eileen: ugh, outta space on the CI box again [23:20:56] holler if you see smoke :) [23:21:09] ejegg: boo [23:21:21] ejegg: well I can kick it off without that patch & we can do an update [23:21:56] nuking the workspace... [23:22:17] :-) [23:22:46] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/427319 (owner: Eileen) [23:23:22] dang I'm so tempted to v+2 it just to get going [23:24:04] (CR) Ejegg: [C: -1] "Still has a 2nd array param on that drush_print..." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/427319 (owner: Eileen) [23:25:04] (Merged) jenkins-bot: Fix totals field to be search-range searchable [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/428538 (owner: Eileen) [23:26:56] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428541 [23:27:05] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428541 (owner: Eileen) [23:27:16] ah, eileen do you need civi up to make sure the deploy is good? [23:27:23] i kilt it [23:28:11] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428541 (owner: Eileen) [23:28:15] cwd no - not until updb finishes [23:29:23] !log civicrm revision changed from b1e7ccfc4d to 347e613aa5, config revision is 07dee62bff [23:29:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:31:39] (PS15) Eileen: Drush script to fill totals fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/427319 [23:32:00] ejegg: I think I got the drush_print now - sigh [23:32:30] that pesky thing! [23:32:39] :-) [23:34:37] (CR) Ejegg: [C: 2] Drush script to fill totals fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/427319 (owner: Eileen) [23:34:44] yay [23:35:02] :) [23:37:55] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428542 [23:44:13] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428542 (owner: Eileen) [23:45:14] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/428542 (owner: Eileen) [23:51:47] !log civicrm revision changed from 347e613aa5 to 142edbb90b, config revision is 07dee62bff [23:51:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log