[00:04:10] I think those fail mails might have been an associated cache clear [00:14:21] relocating [00:23:26] lots of fail mail and some civi1001 cpu warnings [01:21:29] looking [01:38:26] ejegg: I think it was just a passing thing [07:00:19] PROBLEM - check_puppetrun on frdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-server] [07:05:19] RECOVERY - check_puppetrun on frdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:48:15] hm that is weird [14:48:56] Jeff_Green: ^ [14:49:13] cwd: what's that? [14:49:18] unless you happened to be on the computer at 2am [14:49:37] that puppet fail [14:49:40] oh sorry scrollback [14:49:48] not unless I was sleepwalking... [14:50:04] check_puppetrun on frdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[mariadb-server] [14:50:16] puppetfail seems typical, not weird to me! [14:50:39] hehe yeah [14:50:42] my guess is there was some kind of collision, like system apt-update vs puppet apt-update or something, should be able to figure it out from the logs [14:52:17] it seems like it should be ignoring mariadb* [14:52:30] but maybe just a cache update yeah [14:52:51] it's trying to enforce a package lock [16:11:41] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Reading-Infrastructure-Team-Backlog (Kanban): Remove payment methods header image - https://phabricator.wikimedia.org/T211214 (schoenbaechler) p:Triage>Normal [16:17:01] hey fr-tech :) [16:17:03] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Reading-Infrastructure-Team-Backlog (Kanban): Always show banner first in the feed - https://phabricator.wikimedia.org/T211216 (schoenbaechler) p:Triage>Normal [16:17:42] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Reading-Infrastructure-Team-Backlog (Kanban): Always show banner first in the feed - https://phabricator.wikimedia.org/T211216 (schoenbaechler) a:Mholloway>None [16:21:37] Wikimedia-Fundraising, Mobile-Content-Service, Reading-Infrastructure-Team-Backlog, Wikipedia-Android-App-Backlog: Always show banner first in the feed - https://phabricator.wikimedia.org/T211216 (bearND) [16:22:00] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog: Always show banner first in the feed - https://phabricator.wikimedia.org/T211216 (bearND) [16:26:43] (CR) Jgleeson: [C: 2] "Looks good to me" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476786 (owner: Ejegg) [16:28:11] (Merged) jenkins-bot: Add fallback from Kabyle to Arabic for Ingenico [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476786 (owner: Ejegg) [16:32:36] fr-tech any news for scrum of scrums? [16:33:11] not from me ejegg [16:33:36] ejegg: nothing here, thanks! [16:34:48] cstone: ^ "scrum-of-scrums" is a weekly short meeting of representatives from different engineering teams. https://www.mediawiki.org/wiki/Scrum_of_scrums [16:35:01] jgleeson: how was the trip? How's the housing going? [16:37:05] hey hey AndyRussG, the flight was delayed 3 hours due to thunder and lightening storms (the same ones I was enjoying the night before on standup) so we got back last night at 2:30am! We're at my parents now for a couple of days until we get the keys for our new place and then we're gonna move our stuff in over the weekend so it's looking good good :) [16:37:28] lightning* [16:41:23] jgleeson: wow! must be excited to get settled... hope u can get some rest, too! [16:42:10] yeah it's a big relief to be home [16:43:35] when we checked it was pretty expensive to have the baby over there so it would of difficult on a number of fronts if we had to stay. We were pretty lucky all things considered [16:43:57] would have been difficult [16:44:57] how are things with you AndyRussG ? [16:49:31] jgleeson: heh okok, thanks... Cecilia forgot her jacket at home this morning, and it's pretty chilly today, so I had to come back to get it, then go back to the school to drop it off [16:50:26] I got to take a nap in the bus on the way back, though, which was nice :) [16:55:46] ha :D [17:32:35] fr-tech, be a few minutes late to standup. Need to switch connections as I can't get on hangouts for some reason over my mobile data :\ [17:37:25] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog: figure out oanda google docs request issues - https://phabricator.wikimedia.org/T211227 (DStrine) [17:41:31] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog: figure out oanda google docs request issues - https://phabricator.wikimedia.org/T211227 (DStrine) p:Triage>High [18:00:30] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Reading-Infrastructure-Team-Backlog (Kanban): Remove payment methods header image - https://phabricator.wikimedia.org/T211214 (bearND) a:bearND [18:23:57] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog: Always show banner first in the feed - https://phabricator.wikimedia.org/T211216 (Mholloway) > bearND removed a project: Reading-Infrastructure-Team-Backlog (Kanban). > Herald added a project: Reading-Infrastructure-Team-B... [18:30:38] Fundraising-Backlog: Need automated Endowment receipts in Civi - https://phabricator.wikimedia.org/T211230 (MNoorWMF) [18:48:42] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban): Remove payment methods header image - https://phabricator.wikimedia.org/T211214 (bearND) @schoenbaechler We can deploy this change today. [19:26:17] Fundraising-Backlog: Need automated Endowment receipts in Civi - https://phabricator.wikimedia.org/T211230 (CCogdill_WMF) Working on copy modifications here, will update when approved by the powers that be: https://docs.google.com/document/d/1TcUUYxpGZFlLoxs27gRNZzi-nnx-vyMkMhBu-pKet7o/edit?ts=5c0818e8# [19:26:42] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog: Need automated Endowment receipts in Civi - https://phabricator.wikimedia.org/T211230 (DStrine) [19:26:45] Fundraising-Backlog: donation banners persist for Vivaldi? - https://phabricator.wikimedia.org/T211232 (MBeat33) [19:27:36] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog: Need automated Endowment receipts in Civi - https://phabricator.wikimedia.org/T211230 (CCogdill_WMF) There is a question re: how much work, or how much MORE work, would it be to have the email come from Jimmy, while still keepin... [19:38:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 2594 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 1 hours - memory use is 3.53M (peak 15.51M, 0.08% of max, fragmentation 1.93%), connected_slaves is 2, donations is 39, jobs is 0, jobs-adyen is 0, jobs-paypal is 50, payments-init is 125, pending is 103, recurring is 39, refund is 0, unsubscribe is 66 [19:43:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 3441 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 2 hours - memory use is 4.08M (peak 15.51M, 0.09% of max, fragmentation 1.79%), connected_slaves is 2, donations is 329, jobs is 0, jobs-adyen is 0, jobs-paypal is 33, payments-init is 226, pending is 30, recurring is 19, refund is 0, unsubscribe is 75 [19:48:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 3843 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 2 hours - memory use is 4.09M (peak 15.51M, 0.09% of max, fragmentation 1.77%), connected_slaves is 2, donations is 47, jobs is 0, jobs-adyen is 0, jobs-paypal is 40, payments-init is 134, pending is 33, recurring is 30, refund is 0, unsubscribe is 82 [19:53:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 4220 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 161 days 2 hours - memory use is 4.53M (peak 15.51M, 0.09% of max, fragmentation 1.65%), connected_slaves is 2, donations is 375, jobs is 0, jobs-adyen is 0, jobs-paypal is 55, payments-init is 96, pending is 0, recurring is 42, refund is 0, unsubscribe is 95 [19:58:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 4573 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 161 days 2 hours - memory use is 4.68M (peak 15.51M, 0.10% of max, fragmentation 1.71%), connected_slaves is 2, donations is 38, jobs is 0, jobs-adyen is 1, jobs-paypal is 48, payments-init is 254, pending is 92, recurring is 52, refund is 0, unsubscribe is 99 [20:03:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 5056 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 2 hours - memory use is 5.21M (peak 15.51M, 0.10% of max, fragmentation 1.63%), connected_slaves is 2, donations is 347, jobs is 0, jobs-adyen is 0, jobs-paypal is 30, payments-init is 218, pending is 99, recurring is 19, refund is 0, unsubscribe is 7 [20:08:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 5606 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 2 hours - memory use is 5.40M (peak 15.51M, 0.11% of max, fragmentation 1.64%), connected_slaves is 2, donations is 54, jobs is 0, jobs-adyen is 0, jobs-paypal is 57, payments-init is 136, pending is 98, recurring is 24, refund is 0, unsubscribe is 14 [20:13:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6133 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 2 hours - memory use is 6.01M (peak 15.51M, 0.11% of max, fragmentation 1.55%), connected_slaves is 2, donations is 363, jobs is 0, jobs-adyen is 0, jobs-paypal is 33, payments-init is 331, pending is 93, recurring is 32, refund is 0, unsubscribe is 20 [20:18:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6622 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 2 hours - memory use is 6.19M (peak 15.51M, 0.11% of max, fragmentation 1.51%), connected_slaves is 2, donations is 56, jobs is 0, jobs-adyen is 0, jobs-paypal is 52, payments-init is 181, pending is 109, recurring is 41, refund is 0, unsubscribe is 27 [20:23:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 6977 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 161 days 2 hours - memory use is 6.44M (peak 15.51M, 0.12% of max, fragmentation 1.52%), connected_slaves is 2, donations is 369, jobs is 0, jobs-adyen is 0, jobs-paypal is 48, payments-init is 72, pending is 0, recurring is 10, refund is 0, unsubscribe is 39 [20:28:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7377 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 2 hours - memory use is 6.53M (peak 15.51M, 0.12% of max, fragmentation 1.56%), connected_slaves is 2, donations is 33, jobs is 0, jobs-adyen is 0, jobs-paypal is 41, payments-init is 229, pending is 90, recurring is 26, refund is 0, unsubscribe is 43 [20:33:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 7856 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 2 hours - memory use is 7.17M (peak 15.51M, 0.13% of max, fragmentation 1.47%), connected_slaves is 2, donations is 339, jobs is 0, jobs-adyen is 0, jobs-paypal is 57, payments-init is 236, pending is 112, recurring is 39, refund is 0, unsubscribe is 57 [20:38:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 8385 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 2 hours - memory use is 7.24M (peak 15.51M, 0.13% of max, fragmentation 1.52%), connected_slaves is 2, donations is 49, jobs is 0, jobs-adyen is 0, jobs-paypal is 43, payments-init is 113, pending is 82, recurring is 51, refund is 0, unsubscribe is 67 [20:43:10] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog: figure out oanda google docs request issues - https://phabricator.wikimedia.org/T211227 (DStrine) This seems to have been a false alarm [20:43:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 8866 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 3 hours - memory use is 7.81M (peak 15.51M, 0.14% of max, fragmentation 1.45%), connected_slaves is 2, donations is 355, jobs is 0, jobs-adyen is 0, jobs-paypal is 37, payments-init is 268, pending is 113, recurring is 15, refund is 0, unsubscribe is 70 [20:43:19] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog: figure out oanda google docs request issues - https://phabricator.wikimedia.org/T211227 (DStrine) Open>declined [20:44:12] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog: URGENT: Civi Search Timing out - https://phabricator.wikimedia.org/T211062 (DStrine) I have asked Pats to ask Jasmeet for this data instead of using Civi. I'm closing this topic. [20:44:19] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog: URGENT: Civi Search Timing out - https://phabricator.wikimedia.org/T211062 (DStrine) Open>Resolved [20:48:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 9254 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 3 hours - memory use is 7.80M (peak 15.51M, 0.14% of max, fragmentation 1.48%), connected_slaves is 2, donations is 35, jobs is 0, jobs-adyen is 0, jobs-paypal is 46, payments-init is 163, pending is 90, recurring is 23, refund is 0, unsubscribe is 82 [20:49:38] (PS1) Ejegg: Cap session velocity score at 1000 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/477853 [20:50:56] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: payments-antifraud is 9254 2000 - REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 161 days 3 hours - memory use is 7.80M (peak 15.51M, 0.14% of max, fragmentation 1.48%), connected_slaves is 2, donations is 35, jobs is 0, jobs-adyen is 0, jobs-paypal is 46, payments-init is 163, pending is 90, recurring is 23, refund is 0, unsubscribe is 82 [20:50:56] s known and under investigation [21:29:53] (CR) Eileen: [C: 2] Cap session velocity score at 1000 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/477853 (owner: Ejegg) [21:32:08] (Abandoned) Ejegg: Update DonationInterface submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/471881 (owner: Ejegg) [21:45:18] (Merged) jenkins-bot: Cap session velocity score at 1000 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/477853 (owner: Ejegg) [21:50:02] (PS4) XenoRyet: Ingenico Full Redirect [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/476766 (https://phabricator.wikimedia.org/T209413) [21:57:43] (PS1) Ejegg: Send to damaged store on JsonException [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/477915 [22:05:45] Fundraising-Backlog: Banners allow Other Amount=0 if fee selected - https://phabricator.wikimedia.org/T211249 (MBeat33) [22:23:09] Fundraising-Backlog: Banners allow Other Amount=0 if fee selected - https://phabricator.wikimedia.org/T211249 (Platonides) You don't even need to write a 0. * [Visit the banner](https://en.wikipedia.org/wiki/NASA?banner=B1819_120422_en4C_dsk_p1_lg_ptf_cnt&force=1&country=US) * Select an amount (eg. $3). * A... [22:24:41] eileen: so, an empty string is NOT valid json after all [22:24:58] ejegg: hmm ok [22:25:09] so how does that change your decision on what to do? [22:26:02] I wouldn't want to try to push a change upstream to php-queue that accepts invalid json [22:26:21] maybe go with the damaged store change? [22:26:27] i.e. https://gerrit.wikimedia.org/r/477915 [22:26:57] so, the valid JSON for the empty string is "" [22:27:08] hmm, but... is that what's in redis? [22:27:13] or is redis quoting things for me? [22:29:09] ok, redis is quoting that [22:29:23] so they really are empty strings [22:37:53] so... how are we pushing these? [22:42:16] ejegg: I feel like I want to approve that & we can get the queue down & maybe mull it a little more [22:43:22] sounds good to me eileen [22:43:41] (CR) Eileen: [C: 2] Send to damaged store on JsonException [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/477915 (owner: Ejegg) [22:44:23] so we SHOULD be checking for errors on push... [22:44:35] (Merged) jenkins-bot: Send to damaged store on JsonException [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/477915 (owner: Ejegg) [22:44:59] would be really freaking nice if JSON_THROW_ON_ERROR were available before php 7.3 [22:45:11] or, y'know, the default behavior [22:46:43] ejegg: so where do we push that in? [22:48:07] I need to push a new version to packagist [22:48:11] then pull the update in CRM [22:48:33] last tag was v0.5.8.3 [22:48:43] so I'll just do the most minor of increments [22:48:46] to v0.5.8.4 [22:49:27] ok, pushed that tag [22:50:09] now updating at https://packagist.org/packages/wikimedia/smash-pig [22:50:21] cstone: ever managed a composer package? [22:53:14] cstone: if you make yourself a packagist account, I'll add you as an admin on our packages [22:53:30] XenoRyet: oh hey, looks like you're not a smash-pig admin? [22:53:41] XenoRyet: do you have a packagist account? [22:54:17] Let me find out. [22:55:03] It would appear so. Same username as my nick [22:55:21] Fundraising Sprint Window dressing is mostly olive oil, Fundraising-Backlog: Need automated Endowment receipts in Civi - https://phabricator.wikimedia.org/T211230 (CCogdill_WMF) Okay, the copy at the top of this doc is finalized: https://docs.google.com/document/d/1TcUUYxpGZFlLoxs27gRNZzi-nnx-vyMkMhBu-pK... [22:57:01] ejegg: I just made an account, cstone for my username [22:57:20] great! adding you both [22:59:39] Cool, thanks [23:16:29] (PS1) Ejegg: Update SmashPig library [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/477930 [23:16:33] (CR) Ejegg: [C: 2] Update SmashPig library [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/477930 (owner: Ejegg) [23:17:36] (PS1) Ejegg: Update smash-pig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/477931 [23:17:41] (CR) Ejegg: [C: 2] Update smash-pig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/477931 (owner: Ejegg) [23:21:13] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/477932 [23:21:23] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/477932 (owner: Ejegg) [23:21:49] (Merged) jenkins-bot: Update SmashPig library [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/477930 (owner: Ejegg) [23:22:19] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/477932 (owner: Ejegg) [23:23:48] (CR) Ejegg: [V: 2 C: 2] Update smash-pig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/477931 (owner: Ejegg) [23:25:11] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/477933 [23:25:31] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/477933 (owner: Ejegg) [23:26:06] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban): Remove payment methods header image - https://phabricator.wikimedia.org/T211214 (schoenbaechler) Thanks @bearND, let’s do it! cc @spatton [23:26:22] (Merged) jenkins-bot: Update smash-pig [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/477931 (owner: Ejegg) [23:27:57] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/477936 [23:28:06] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/477936 (owner: Ejegg) [23:28:34] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/477933 (owner: Ejegg) [23:28:48] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/477936 (owner: Ejegg) [23:32:11] !log turned off fundraising queue jobs for base queue consumer logic update [23:32:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:33:15] !log updated fundraising CiviCRM from e757753a46 to 3e5d74f17e [23:33:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:34:03] ok, pending qc still looks like it works [23:34:39] fr-tech sorry for the failstorm! seem to be getting past the blanks at least, though [23:35:41] ejegg: the "** PROBLEM alert - frqueue1001/check_redis is CRITICAL **" stuff? [23:36:24] AndyRussG: I think there are a few hundred 'bad message format' emails incoming [23:37:09] "Fail Mail (civi1001) run-job: Fredge Multiqueue Consumer failed with code 1" ? [23:37:27] AndyRussG: that was the previous one, when it was bailing on the first bad message [23:37:33] hummm [23:37:40] there's a new crop of fredge_batch_process ones just coming in now [23:38:03] Ooh, but it's smart enough to batch those [23:38:07] OK, not so horrible [23:38:14] Ah I got one now [23:38:38] ok, will turn the queues back on [23:38:49] lmk if I can help w/ anything on that end of stuff... now that my Vagrant works... :) [23:38:50] Wikimedia-Fundraising, Mobile-Content-Service, Wikipedia-Android-App-Backlog, Patch-For-Review, Reading-Infrastructure-Team-Backlog (Kanban): Remove payment methods header image - https://phabricator.wikimedia.org/T211214 (Mholloway) The response change is now live. [23:39:51] ejegg: what’s tha new failmail? - that’s just the expected new error? [23:40:06] eileen: yep, that's the good behavior [23:40:16] phew [23:40:20] !log re-enabled fundraising queue consumer jobs [23:40:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:40:51] I was afraid for a second that trying to store null in the damaged table would blow up all over again, but that bit's resiliant [23:43:39] ejegg: I’m really happy to jump on a call like that if we hit something similar again - even if it doesn’t co-incide with tech-talk :-) [23:44:02] it felt like it might not have sped up your solving but it meant that once you had a patch the review process was quick :p-) [23:45:00] definitely! I'm always up for live-action problem-solving [23:48:12] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 161 days 6 hours - memory use is 2.34M (peak 18.28M, 0.08% of max, fragmentation 2.86%), connected_slaves is 2, donations is 42, jobs is 0, jobs-adyen is 0, jobs-paypal is 36, payments-antifraud is 130, payments-init is 150, pending is 81, recurring is 47, refund is 1, unsubscribe is 75 [23:50:07] !log updated payments-wiki from 20595cca97 to b99cd0816e [23:50:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log