[01:00:16] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1261 [01:03:49] smash [01:04:31] 2001 is asleep on the job again [01:05:16] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1560 [01:10:16] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1860 [01:15:16] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2160 [01:20:16] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2460 [01:25:16] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2760 [01:30:16] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3060 [01:35:16] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3361 [01:40:16] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 19162 Threads: 2 Questions: 364575 Slow queries: 627 Opens: 597 Flush tables: 1 Open tables: 243 Queries per second avg: 19.025 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [02:05:17] cwd I have been hammering staging a bit & while I can't see why hammering the staging db would affect replication (not the copy of the live db on staging) I am wondering [02:05:28] since I'm not sure what other load is happening - perhaps a test? [02:05:54] no - that was hours earlier [02:09:45] https://civicrm.org/blog/colemanw/announcing-civicrm-version-01beta0 [03:19:48] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Create lists of CiviCRM groups to allow MG & DS to review with a view to tidy up - https://phabricator.wikimedia.org/T174407#3606647 (Eileenmcnaughton) Sent email to DS/ MG/FT-... [03:39:33] eileen: no idea, it seems to be back to the same pattern as before https://grafana.wikimedia.org/dashboard/db/frdb2001?orgId=1 [03:39:45] jeff silenced icinga for tonight [07:13:04] (PS1) Eileen: Update omnimail to validate dates & silverpop package to validate returning xml [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/377958 (https://phabricator.wikimedia.org/T175394) [07:37:13] https://uk.reuters.com/article/us-google-india/google-to-launch-mobile-payment-service-in-india-report-idUKKCN1BP0GP [12:55:06] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1366 [13:00:15] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1667 [13:05:15] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1967 [13:10:15] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2266 [13:15:15] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2566 [13:20:15] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2867 [13:25:15] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3166 [13:30:15] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3467 [13:35:15] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 62062 Threads: 1 Questions: 1949816 Slow queries: 2747 Opens: 1623 Flush tables: 1 Open tables: 577 Queries per second avg: 31.417 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [13:59:00] hey cwd [14:39:01] mepps: mornin! [14:39:31] how's it going? [14:42:22] not too bad and you? [14:43:43] got queue metrics in grafana https://grafana.wikimedia.org/dashboard/db/fundraising-redis?orgId=1 [14:45:13] nice! [14:45:27] i'm doing good [14:45:37] a lot of info there! [14:51:58] Jeff_Green: anything jump out at you this morning? [14:52:40] not really, I'm going through the general query log for frdb2001 to see if I can find anything obvious [14:52:55] mepps: here are some charts that tell a funny story about this replag https://grafana.wikimedia.org/dashboard/db/frdb2001?orgId=1 [14:53:10] frdb2001 seems to just go to sleep sometimes [14:53:12] oh and this seemed sort of related https://bugs.mysql.com/bug.php?id=84192 [14:53:58] i see a lot of "COMMIT /* implicit, from Xid_log_event */" right after the logjam clears [14:57:37] wow yeah [14:58:26] that's one sad bug report [14:58:47] indeed [14:59:04] It's broken.. The Thing. [15:03:42] sigh cwd, the colors... [15:03:59] but yes i see what you mean [15:05:22] the colors seems like a basic ass feature that is missing [15:05:41] also when you click one or a couple metrics on a chart [15:05:47] that should go in the url somehow so you can link it [15:05:54] let's fork grafana [15:07:14] https://github.com/grafana/grafana/issues/3250 [15:08:30] can we keep frdb2001 awake with some silly little recurring job? [15:09:15] https://github.com/grafana/grafana/pull/9059 [15:09:56] that PR looks like it got close cwd [15:11:13] ejegg: well it doesn't really go to sleep, but the mysql process appears pretty much locked up during that time [15:11:29] mepps: haha awesome [15:11:32] good find [15:12:44] huh, weird. [15:13:10] also morning ejegg! [15:13:27] hi mepps! [15:13:39] anything I should be reviewing? [15:14:01] yes did you see i made comments here: https://gerrit.wikimedia.org/r/#/c/366167/ [15:14:08] looking now [15:14:42] DI and i are still not best friends so if you want to hop on a call to discuss that work, that might help us close it out [15:20:44] (CR) Ejegg: WIP getHostedCheckoutStatus (2 comments) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/366167 (https://phabricator.wikimedia.org/T163948) (owner: Ejegg) [15:26:56] (PS1) Ejegg: WIP test processor error in SET_PAYMENT [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378047 [15:28:48] (CR) jerkins-bot: [V: -1] WIP test processor error in SET_PAYMENT [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/378047 (owner: Ejegg) [15:50:32] ejegg let me know ^^ [15:51:05] oh sure, a call would be great. When [15:51:09] 's good for you? [15:51:31] I'm just rebasing an old thing or two, be ready in 10 min [15:51:37] ejegg that sounds good to me [15:51:46] cool [15:55:24] (PS11) Ejegg: Truncate referrers when hosts match a pattern [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/232218 (https://phabricator.wikimedia.org/T110564) [15:58:01] (PS2) Ejegg: Some notes on pushing gateway classes down to SmashPig [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/364531 [16:03:33] (PS3) Ejegg: Remove some unused and redundant i18n keys [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/288527 [16:05:26] ejegg let me know when you're ready [16:09:19] (PS10) Ejegg: Use ct_id to find completed, avoid race [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312563 (https://phabricator.wikimedia.org/T143945) [16:09:57] mepps ok, hopping in the call [16:10:13] (CR) jerkins-bot: [V: -1] Use ct_id to find completed, avoid race [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312563 (https://phabricator.wikimedia.org/T143945) (owner: Ejegg) [16:34:05] (PS11) Ejegg: Use ct_id to find completed, avoid race [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312563 (https://phabricator.wikimedia.org/T143945) [16:35:05] (CR) jerkins-bot: [V: -1] Use ct_id to find completed, avoid race [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312563 (https://phabricator.wikimedia.org/T143945) (owner: Ejegg) [16:37:41] (PS12) Ejegg: Use ct_id to find completed, avoid race [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312563 (https://phabricator.wikimedia.org/T143945) [16:43:58] (PS7) Ejegg: Drop autoincrement ID and FKs on group_contact_cache [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/374588 (https://phabricator.wikimedia.org/T174404) [17:00:27] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Unplanned-Sprint-Work: Investigate email: BH data storage/transfer issue for iPad donations - https://phabricator.wikimedia.org/T174719#3571255 (AndyRussG) Patch for r... [17:00:48] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate email: BH data storage/transfer issue for iPad donations - https://phabricator.wikimedia.org/T174719#3608449 (AndyRussG) [17:26:43] (PS8) Ejegg: WIP getHostedCheckoutStatus [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/366167 (https://phabricator.wikimedia.org/T163948) [17:29:06] mepps ^^^ gets us as far as the SET_PAYMENT call, which is still specific to the old XML integration [17:31:10] Should I fold you 'WIP try approvePayment' patch into this one so we can successfully get through the Confirm_CreditCard? [17:31:38] Or will you be able to usefully test stuff with just that? [17:33:29] (PS8) Ejegg: Maintenance script to create all db tables [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312577 [17:33:50] (PS9) Ejegg: Rename some impressionDiet variables [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/319003 (https://phabricator.wikimedia.org/T121178) [17:33:57] (PS5) Ejegg: Undo 'require' debug mode hack [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/319259 [17:34:31] (CR) jerkins-bot: [V: -1] Maintenance script to create all db tables [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312577 (owner: Ejegg) [17:34:39] (PS4) Ejegg: Drop some unused indexes on silverpop export table [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/319393 [17:35:41] (PS9) Ejegg: Maintenance script to create all db tables [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/312577 [17:38:42] (PS12) Ejegg: Capture Adyen payments missing pending messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/319489 (https://phabricator.wikimedia.org/T149861) [17:40:21] (PS3) Ejegg: [WIP] Fail the message if a currency or amount are given alone, rather than as a pair. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/164499 (owner: Awight) [17:42:28] (Abandoned) Ejegg: WIP add remove button to widgets [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/197360 (owner: Ssmith) [17:44:16] (Abandoned) Ejegg: [WIP] Always log the message that would be sent to Civi [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/178981 (owner: Awight) [17:46:17] (PS2) Ejegg: Ignore invalid country codes from query string [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/324789 (https://phabricator.wikimedia.org/T152026) [17:46:20] hey ejegg i'm still seeing the same error with your new patch--should i be? [17:49:11] (PS4) Ejegg: TEST: do we need the refresh? [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/359233 [17:52:03] (CR) jerkins-bot: [V: -1] Ignore invalid country codes from query string [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/324789 (https://phabricator.wikimedia.org/T152026) (owner: Ejegg) [17:52:11] (CR) jerkins-bot: [V: -1] [WIP] Fail the message if a currency or amount are given alone, rather than as a pair. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/164499 (owner: Awight) [17:52:45] mepps that's probably because it's trying to use the old logic for SET_PAYMENT [17:53:01] do you see it getting to the point of calling that, though? [17:53:19] see my questions in the backscroll (13:29) [17:57:06] (PS3) Ejegg: Ignore invalid country codes from query string [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/324789 (https://phabricator.wikimedia.org/T152026) [17:59:18] (Abandoned) Ejegg: [WIP] Fix totally wrong language things [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/205348 (owner: Awight) [18:00:49] (PS5) Ejegg: Remove unnecessary trigger refresh [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/359233 [18:03:56] ejegg should i be seeing this in the logs: Sep 14 14:03:16 uranus ingenico_gateway: 3078:3078.3 We don't have an order status after doing a GET_ORDERSTATUS. [18:04:23] mepps oh shoot, I thought we'd have that now. lemme see [18:05:52] k, guess I need to do some more work on that [18:05:59] will get back to it after some lunch [18:49:51] so I've got a cute little monitor that's great for keeping up on IRC [18:50:08] but the only problem is it totally freezes up my computer when I unplug it [18:50:35] thinking this might fix it: https://patchwork.kernel.org/patch/9759479/ [18:51:05] but it's been a while since I compiled my own kernel [18:57:48] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 2 others: Investigate email: BH data storage/transfer issue for iPad donations - https://phabricator.wikimedia.org/T174719#3608879 (AndyRussG) **Summary:** * I've been... [18:58:00] nice, debian testing has a bleeding edge kernel available with the patch already in it [18:58:06] seems to work! [19:06:47] ejegg how do you have phpstorm configured to run tests for DI? [19:07:16] i've just been running them in the terminal because i keep getting an error that it can't find the DITestCase in phpstorm [19:11:59] mepps yeah, I've been running via terminal too [19:12:31] ahh okay [19:13:14] i have this little shortcut script: [19:13:16] https://gist.github.com/ejegg/48706966ba19b0c173406afb2cb811de [19:13:52] in my ~/bin folder (on my $PATH) [19:14:09] so 'diphpunit' from any dir runs all the DonationInterface tests [19:14:34] and 'diphpunit GlobalCollect' runs all the tests with @group=GlobalCollect [19:21:07] cool! [19:24:08] ok, got the monitor working ok... back to business! [19:25:40] hmph, phpstorm's latest version fixed the 'navigate to file' bug, but now the indexing is flaky [19:25:55] gotta regularly choose 'invalidate caches and restart' [19:26:07] the existence of that menu item seems like a failure [20:07:31] (PS9) Ejegg: WIP getHostedCheckoutStatus [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/366167 (https://phabricator.wikimedia.org/T163948) [20:08:17] mepps ok, ^^^ really does get us up to the point of running the SET_PAYMENT call [20:08:38] great ejegg! [20:08:59] let me know what you think of the comment where I flatten the json response into a k -> v array [20:16:26] ejegg i'm getting a fatal error [20:16:41] where? [20:16:43] PHP Fatal error: Cannot redeclare IngenicoAdapter::processDonorReturn() in /var/www/fr-tech/mediawiki_payments/extensions/DonationInterface/ingenico_gateway/ingenico.adapter.php on line 247 [20:16:55] when i tried to look at it in the browser [20:17:27] wait, how is it trying to redeclare that? [20:17:46] my file of that name ends at line 247 [20:19:49] ejegg huh wierd [20:20:09] what does git diff say? [20:20:23] did you have to do some kind of merge when you pulled that change down? [20:20:42] ejegg i did not but i'll try starting over [20:21:03] cwd did you still want to talk today? it seems like eileen isn't around and XenoRyet is out sick but i'm happy to chat [20:21:33] mepps: talking w/ jeff in privchan right now, happy to do it there [20:23:08] okay ejegg i'm good, it's weird, i hadn't changed anything on the branch but i did pull down the patch diferently [20:23:25] huh, that is weird [20:23:48] mepps also, it occurs to me that we can probably just un-skip a bunch of the currently skipped tests [20:24:07] ejgg yeah i'm doing that but there are still some issues to be worked out [20:24:17] ah, cool [20:24:25] ejegg nice i get the donate error page now! [20:24:34] well, that's progress! [20:24:57] the log should complain about 'IPADDRESS' not being in the txn structure [20:25:11] since it's part of the parent SET_PAYMENT call, I think [20:25:55] hey [20:26:00] hi eileen! [20:26:07] a bit late to talk to the duck [20:27:04] (Abandoned) Ejegg: [WIP] SourceFields experiment [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/311884 (owner: Awight) [20:28:42] hey eileen, we still could, no call has happened yet [20:29:21] i'm getting this " ingenico_gateway: 3079:3079.5 SET_PAYMENT couldn't communicate properly!" but that seems appropriate [20:29:42] yeah - let's talk replag [20:29:43] (Abandoned) Ejegg: WIP issuer patch [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/281871 (owner: Awight) [20:30:06] mepps yep, that's where it should get to! [20:30:29] hey cwd we're in the call [20:30:45] cool, brt [20:35:34] (CR) Ejegg: "I like the TestConfiguration clean up. I guess it doesn't matter if we stuff the testing classes in the autoload array, even for prod? I w" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/307542 (owner: Awight)