[01:06:38] PROBLEM - check_minfraud_secondary on payments1002 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [01:06:39] PROBLEM - check_minfraud_secondary on payments1004 is CRITICAL: CRITICAL - Socket timeout after 61 seconds [01:10:18] RECOVERY - check_minfraud_secondary on payments1002 is OK: HTTP OK: Status line output matched 200,301,302 - 385 bytes in 0.247 second response time [01:10:19] RECOVERY - check_minfraud_secondary on payments1004 is OK: HTTP OK: Status line output matched 200,301,302 - 385 bytes in 0.249 second response time [02:18:49] (PS4) Ejegg: Update phpunit to use 5.7 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363291 (owner: Eileen) [02:21:18] (CR) Ejegg: "Composer has learned some new tricks! I got the lock file in PS4 by running "composer update phpseclib/phpseclib phpunit/*"" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363291 (owner: Eileen) [02:21:49] (CR) Ejegg: "After we merge this and your new extension I'll do another patch updating the SmashPig version and usage." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363291 (owner: Eileen) [02:26:28] (PS5) Ejegg: Update phpunit to use 5.7 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363291 (owner: Eileen) [02:26:50] (CR) Ejegg: [C: 2] Update phpunit to use 5.7 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363291 (owner: Eileen) [02:33:09] (Merged) jenkins-bot: Update phpunit to use 5.7 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363291 (owner: Eileen) [14:15:11] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 7 minutes ago with 11 failures. Failed resources (up to 3 shown) [14:20:11] PROBLEM - check_puppetrun on rigel is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 9 minutes ago with 15 failures. Failed resources (up to 3 shown) [14:20:12] RECOVERY - check_puppetrun on mintaka is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:25:11] RECOVERY - check_puppetrun on rigel is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:50:15] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [14:55:15] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [15:00:05] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 148585 [15:00:55] heh sorry about ^ [15:01:04] i forgot to restart this the other night [15:02:08] we _should_ have been paged like 36 hours ago [15:02:19] not sure what happened there [15:05:05] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 148885 [15:05:49] https://gfycat.com/BlackandwhiteScaredAstarte [15:06:53] ^ Just some random brain bleach to start the day ;) [15:10:06] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 149185 [15:11:56] ACKNOWLEDGEMENT - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 149185 Jeff_Green known and were watching it [17:00:32] fr-tech: Safety Tips for the Post-Nuclear Existence [17:00:32] Tip #1: How to tell when you are dead. [17:00:32] 1. Little things start bothering you: little things like worms, [17:00:32] bugs, ants. [17:00:32] 2. Something is missing in your personal relationships. [17:00:33] 3. Your dog becomes overly affectionate. [17:00:33] 4. You have a hard time getting a waiter. [17:00:34] 5. Exotic birds flock around you. [17:00:34] 6. People ignore you at parties. [17:00:35] 7. You have a hard time getting up in the morning. [17:00:35] 8. You no longer get off on cocaine. [17:00:36] -- discuss. [17:00:51] yikes [17:37:39] Fundraising Sprint Murphy's Lawyer, Fundraising-Backlog, WMF-NDA: Amazon (aaaaaargh!) audit report downloads failing, something about an IP address - https://phabricator.wikimedia.org/T169805#3416111 (Ejegg) OK, looks like we were eventually able to download the reports. @Jgreen and @Cdentinger, I g... [17:45:10] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 1294247 Threads: 1 Questions: 28809637 Slow queries: 7150 Opens: 8010 Flush tables: 1 Open tables: 497 Queries per second avg: 22.259 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [17:54:56] (PS1) Ejegg: PHPunit 5.7, not 5.6 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363854 [17:55:55] (CR) Ejegg: [C: 2] PHPunit 5.7, not 5.6 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363854 (owner: Ejegg) [17:59:48] (PS25) Ejegg: Initial commit, omnimail extension & extendedmailingreport [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/360610 (https://phabricator.wikimedia.org/T169593) (owner: Eileen) [18:04:46] yikes indeed [18:06:41] (Merged) jenkins-bot: PHPunit 5.7, not 5.6 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363854 (owner: Ejegg) [18:10:37] (CR) jerkins-bot: [V: -1] Initial commit, omnimail extension & extendedmailingreport [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/360610 (https://phabricator.wikimedia.org/T169593) (owner: Eileen) [18:17:54] ejegg is there a reason we're not using: https://github.com/Ingenico-ePayments/connect-sdk-php [18:20:48] mepps I was going back and forth on that [18:22:03] what were your thoughts? i ask because the documentation references it pretty heavily [18:22:12] If we do things our own way, we can use consistent low-level stuff, like cURL and logging [18:22:56] also, if we use a different SDK for each processor, there's the possibility of conflicting dependencies [18:23:18] fundraising-tech-ops, Mail, Operations: (re)move problemsdonating aliases - https://phabricator.wikimedia.org/T127488#3416411 (eross) @Dzahn From the ticket on Zendesk it stated problemsdonate@ but I can change if it needs to be problemsdonating@ however, problemsdonating@ is already created as an u... [18:23:20] yeah i can see that [18:23:38] That said, we're currently using a forked version of Amazon's PHP SDK [18:23:46] Fundraising Sprint Murphy's Lawyer, Fundraising-Backlog, WMF-NDA: Amazon (aaaaaargh!) audit report downloads failing, something about an IP address - https://phabricator.wikimedia.org/T169805#3416423 (Jgreen) >>! In T169805#3416111, @Ejegg wrote: > OK, looks like we were eventually able to download t... [18:25:06] ah, hmm, i can see it both ways, being dependent on outside maintenance can be tricky but they may also catch things before we do--i also feel a little like i'm reinventing the wheel [18:30:57] mepps yeah... my test was whether the signature generation was going to be a huge pain to recreate [18:31:23] and when that wasn't so bad, it looked like the rest was just setting arrays of parameters and parsing JSON coming back [19:24:15] ejegg do you want to check in a bit before standup? [19:26:10] sure, give me 5 min and I'll be in queenmary [19:42:50] ejegg let me know if your connection improves [20:22:31] hey dstrine I’ve been off IRC so maybe this is old news… but I’m not able to load the form on donate.wiki [20:22:35] is that expected? [20:23:59] ccogdill: do you mean donate.wikimedia.org [20:24:30] yeah [20:24:40] ok yeah we don't see the form loading [20:24:41] I tried on 2 browsers [20:26:26] ccogdill: we are looking into it [20:26:39] okay thanks [20:27:09] MBeat: are you seeing any reports^^^ [20:27:27] not yet [20:28:03] wow good catch ccogdill [20:28:25] just brainstorming button tests and this is what you get :p [20:29:45] Moska says she was able to 30 mins ago [20:29:47] ccogdill: thanks for the notice. this particular problem is a bit hard to detect automatically. We've haven't nailed down exactly what's happening [20:29:50] if that’s useful info [20:32:41] ccogdill: can she tell us which url she used? [20:32:54] ccogdill is Moska able to see it now (if she's still around)? [20:33:06] I know it was an email link [20:33:09] she can’t see it now [20:33:14] I tested some email links, too [20:34:49] (asking her about the specific link) [20:38:23] here’s the link mepps if it matters. form isn’t loading now [20:38:26] https://donate.wikimedia.org/w/index.php?title=Special:LandingPage&country=US&uselang=%25%25isolang%25%25&utm_medium=email&utm_source=sp%25%25MAILING_ID%25%25&utm_campaign=C1617_Email3&appeal=Appeal-JimmyQuote&hpc=%25%25highest_usd_amount%25%25&contact_id=%25%25ContactID%25%25&link_id=2 [20:38:46] ccogdill yeah we can't see it either, ejegg is on the case [20:39:09] cool, thanks! [20:40:03] good catch ccogdill! [20:42:14] happy it was me and not a donor! that’s the best kind of bug report [20:45:26] Fundraising-Backlog, Wikimedia-Fundraising, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-DonationInterface, and 2 others: Triage accessibility for donor-facing Fundraising components - https://phabricator.wikimedia.org/T87667#3416830 (Pcoombe) [20:45:28] Wikimedia-Fundraising, Accessibility: Examine accessibility of donate.wikimedia.org content - https://phabricator.wikimedia.org/T162934#3416828 (Pcoombe) Open>Resolved I added the improved focus styles, and made it possible to tab past the "Other" input. Feel free to re-open or make another task... [20:46:45] ccogdill not sure if you were updated elsewhere but it should be back to normal [20:47:05] wow that was fast! [20:57:03] ty!! [21:10:01] yeah thanks ccogdill for spotting that! [21:10:17] anytime! [21:10:36] thanks for jumping on it [21:42:40] fundraising-tech-ops, Mail, Operations: (re)move problemsdonating aliases - https://phabricator.wikimedia.org/T127488#3416972 (Dzahn) @eross Thank you Emerauld! Appreciate it. I will remove it on our side and close this ticket here. Yes, the rest can just be up to James A and Fundraising. [22:22:41] (PS1) Umherirrender: build: Updating mediawiki/mediawiki-codesniffer to 0.10.0 [extensions/FundraisingTranslateWorkflow] - https://gerrit.wikimedia.org/r/363956 [22:36:36] fundraising-tech-ops, Mail, Operations: (re)move problemsdonating aliases - https://phabricator.wikimedia.org/T127488#3417112 (Dzahn) Open>Resolved a:Dzahn Removed on ops side. I see that problems.donating , problemdonating and problem.donating and comentarios work in Google. The other... [22:40:52] (PS1) Ejegg: Revert "Revert "Merge branch 'master' into deployment"" [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/363971 [22:40:54] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/363972 [22:53:54] (PS1) Umherirrender: build: Updating mediawiki/mediawiki-codesniffer to 0.10.0 [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/363973 [22:56:58] (CR) Ejegg: "Tested the heck out of this locally, including return processing and repeat return processing for all gateways, and running queue consumer" [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/363972 (owner: Ejegg)