[03:21:32] crud, NO_PAYMENT_PRODUCTS_AVAILABLE via Connect in Chile? [03:24:11] Fundraising Sprint They Live, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Extend deletion to multiple silverpop databases - https://phabricator.wikimedia.org/T205332 (Eileenmcnaughton) @CCogdill_WMF I... [03:27:00] ejegg: is tht the failmail cause? [03:27:27] that's what it looks like [03:27:45] guessing the recurring-ness might have something to do with it [03:28:13] sending an email to PPena to ask if she can confirm what we should have available [03:28:42] ejegg: ok - do we need to take something down for tonight? [03:29:53] let's see if this is just one donor [03:35:52] hmph, that description should be translated [08:31:28] PROBLEM - Host americium is DOWN: PING CRITICAL - Packet loss = 100% [08:37:48] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=11 [critical = 10] [08:42:48] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=15 [critical = 10] [08:47:48] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=20 [critical = 10] [08:52:48] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=27 [critical = 10] [08:57:58] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=35 [critical = 10] [09:02:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=41 [critical = 10] [09:07:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=45 [critical = 10] [09:11:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [09:12:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=47 [critical = 10] [09:16:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=11 [critical = 10] [09:17:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=52 [critical = 10] [09:21:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=13 [critical = 10] [09:21:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [09:22:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=63 [critical = 10] [09:26:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=14 [critical = 10] [09:26:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=11 [critical = 10] [09:27:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=68 [critical = 10] [09:31:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=15 [critical = 10] [09:31:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=12 [critical = 10] [09:32:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=73 [critical = 10] [09:35:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [09:36:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=16 [critical = 10] [09:36:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=14 [critical = 10] [09:37:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=82 [critical = 10] [09:40:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [09:41:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=17 [critical = 10] [09:41:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=15 [critical = 10] [09:42:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=93 [critical = 10] [09:45:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=11 [critical = 10] [09:46:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=19 [critical = 10] [09:46:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=16 [critical = 10] [09:47:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=97 [critical = 10] [09:50:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [09:50:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=12 [critical = 10] [09:51:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=20 [critical = 10] [09:51:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=18 [critical = 10] [09:52:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=102 [critical = 10] [09:55:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [09:55:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=13 [critical = 10] [09:56:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=21 [critical = 10] [09:57:00] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=20 [critical = 10] [09:57:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=108 [critical = 10] [10:00:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=11 [critical = 10] [10:00:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=13 [critical = 10] [10:01:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=22 [critical = 10] [10:01:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=21 [critical = 10] [10:02:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=128 [critical = 10] [10:05:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=12 [critical = 10] [10:05:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=14 [critical = 10] [10:07:07] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=23 [critical = 10] [10:07:07] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=22 [critical = 10] [10:07:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=159 [critical = 10] [10:10:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=12 [critical = 10] [10:10:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=14 [critical = 10] [10:11:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=24 [critical = 10] [10:11:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=24 [critical = 10] [10:12:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=164 [critical = 10] [10:15:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=13 [critical = 10] [10:15:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=15 [critical = 10] [10:16:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=25 [critical = 10] [10:16:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=25 [critical = 10] [10:17:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=169 [critical = 10] [10:20:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=13 [critical = 10] [10:20:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=16 [critical = 10] [10:21:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=26 [critical = 10] [10:21:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=26 [critical = 10] [10:22:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=178 [critical = 10] [10:25:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=13 [critical = 10] [10:25:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=16 [critical = 10] [10:26:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=27 [critical = 10] [10:26:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=27 [critical = 10] [10:27:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=187 [critical = 10] [10:30:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=14 [critical = 10] [10:30:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=17 [critical = 10] [10:31:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=28 [critical = 10] [10:31:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=29 [critical = 10] [10:32:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=192 [critical = 10] [10:35:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=14 [critical = 10] [10:35:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=18 [critical = 10] [10:36:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=29 [critical = 10] [10:36:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=30 [critical = 10] [10:37:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=195 [critical = 10] [10:40:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=15 [critical = 10] [10:40:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=19 [critical = 10] [10:41:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=30 [critical = 10] [10:41:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=31 [critical = 10] [10:42:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=204 [critical = 10] [10:45:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=16 [critical = 10] [10:45:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=20 [critical = 10] [10:46:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=31 [critical = 10] [10:46:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=32 [critical = 10] [10:47:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=209 [critical = 10] [10:50:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=16 [critical = 10] [10:50:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=20 [critical = 10] [10:51:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=32 [critical = 10] [10:51:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=34 [critical = 10] [10:52:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=220 [critical = 10] [10:55:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=17 [critical = 10] [10:55:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=21 [critical = 10] [10:56:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=33 [critical = 10] [10:56:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=35 [critical = 10] [10:57:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=224 [critical = 10] [11:00:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=17 [critical = 10] [11:00:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=21 [critical = 10] [11:01:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=35 [critical = 10] [11:01:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=37 [critical = 10] [11:02:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=236 [critical = 10] [11:05:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=18 [critical = 10] [11:05:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=22 [critical = 10] [11:06:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=36 [critical = 10] [11:06:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=38 [critical = 10] [11:07:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=242 [critical = 10] [11:10:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=18 [critical = 10] [11:10:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=23 [critical = 10] [11:11:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=37 [critical = 10] [11:11:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=39 [critical = 10] [11:12:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=247 [critical = 10] [11:15:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=19 [critical = 10] [11:15:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=23 [critical = 10] [11:16:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=38 [critical = 10] [11:16:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=41 [critical = 10] [11:17:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=252 [critical = 10] [11:20:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=19 [critical = 10] [11:20:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=24 [critical = 10] [11:21:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=39 [critical = 10] [11:21:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=42 [critical = 10] [11:22:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=256 [critical = 10] [11:25:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=20 [critical = 10] [11:25:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=25 [critical = 10] [11:26:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=41 [critical = 10] [11:26:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=43 [critical = 10] [11:27:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=264 [critical = 10] [11:30:10] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=20 [critical = 10] [11:30:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=26 [critical = 10] [11:31:50] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=42 [critical = 10] [11:31:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=44 [critical = 10] [11:32:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=271 [critical = 10] [11:35:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=21 [critical = 10] [11:35:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=27 [critical = 10] [11:36:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=44 [critical = 10] [11:36:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=46 [critical = 10] [11:37:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=275 [critical = 10] [11:40:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=22 [critical = 10] [11:40:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=28 [critical = 10] [11:41:01] Jeff_Green, having some issues [11:41:11] I've mailed a couple of notes [11:41:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=45 [critical = 10] [11:41:40] currently look at switching off the job related to this: [11:41:41] Fail Mail (civi1001) run-job: Banner impressions loader timed out after 10 minutes [11:41:48] as civi1001 is showing 600 in top [11:41:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=48 [critical = 10] [11:42:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=279 [critical = 10] [11:45:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=22 [critical = 10] [11:45:50] PROBLEM - check_rsyslog_backlog on payments1001 is CRITICAL: CRITICAL frlog1001=29 [critical = 10] [11:46:40] PROBLEM - check_rsyslog_backlog on payments1003 is CRITICAL: CRITICAL frlog1001=47 [critical = 10] [11:46:50] PROBLEM - check_rsyslog_backlog on payments1002 is CRITICAL: CRITICAL frlog1001=49 [critical = 10] [11:47:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=284 [critical = 10] [11:48:34] jgleeson: hey, yup I just replied [11:49:17] thanks Jeff_Green, read your reply but struggling to restart rsynclog. I don't have sudo perms [11:49:25] yup [11:49:28] I'm trying sudo service rsyslog restart [11:49:30] i'm looking at it [11:49:35] cool, thanks [11:50:00] PROBLEM - check_rsyslog_backlog on frpig1001 is CRITICAL: CRITICAL frlog1001=23 [critical = 10] [11:50:50] RECOVERY - check_rsyslog_backlog on payments1001 is OK: OK [11:51:06] rsyslog* [11:51:37] there are rsync problems too, it looks like one of the banner loggers fell over [11:51:40] RECOVERY - check_rsyslog_backlog on payments1003 is OK: OK [11:51:46] what the heck happened last night?! [11:51:50] RECOVERY - check_rsyslog_backlog on payments1002 is OK: OK [11:52:08] yeah I'm getting them mixed up lol [11:52:37] I was looking at the log files for that and noticed the extremely high load in tp [11:52:38] top [11:52:49] although I can't see an offending process or CPU load to explain it [11:52:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=283 [critical = 10] [11:52:53] on which host? [11:53:22] civi1001 [11:53:45] load average: 666.33, 653.45, 619.75 [11:53:48] Jeff_Green, ^ [11:53:51] oh really [11:54:05] is that still happening? [11:55:00] RECOVERY - check_rsyslog_backlog on frpig1001 is OK: OK [11:55:05] yes [11:55:11] viewing top now [11:55:15] looking [11:55:25] this doesn't behave like a machine with load >600 [11:56:07] I've never seen load that high! [11:57:39] if it were actually working that hard it should be impossible to do anything b/c there wouldn't be resources for ssh [11:57:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=283 [critical = 10] [11:58:25] it is really slow for me [11:58:39] but yes, I would imagine 600 would mean ground to a halt [11:58:50] RECOVERY - Host americium is UP: PING OK - Packet loss = 0%, RTA = 0.85 ms [11:59:44] well americium had some kind of kernel panic [12:00:34] killing prometheus_node_exporter seems to have coincided with load recovery [12:01:22] I can see it dropping [12:01:24] woah [12:01:27] that was crazy [12:01:44] I wonder if that is the root cause to the banner job timeouts [12:02:12] OH! [12:02:18] ok you just explained it right there [12:02:26] the root cause was americium falling over [12:02:47] americium is the banner logger, it exports its banner log archive by nfs [12:02:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=283 [critical = 10] [12:03:02] ahh [12:03:07] so when it fell over, civi1001 freaked out trying to access that nfs export [12:03:21] nfs is notorious for not handling outages gracefully [12:04:55] hmmm [12:05:09] Jeff_Green, the load on civi1001 is increasing again [12:05:12] 68+ [12:05:47] yup, watching too [12:06:21] something is hammering rsyslog [12:07:50] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=283 [critical = 10] [12:08:40] PROBLEM - check_ipsec on americium is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: civi1001_v4 [12:10:10] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:10] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:11] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:11] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:12] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:12] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:13] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:13] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:14] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:14] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:15] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:15] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:16] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:16] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:17] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:17] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:18] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:18] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:21] hahahah [12:10:30] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:30] :) [12:10:30] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:31] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:31] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:32] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:32] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:33] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:33] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 79.54, 142.74, 354.62 [12:10:51] i think we should just reboot civi1001 [12:12:52] Jeff_Green, do we need to disable paymentswiki first [12:12:54] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=283 [critical = 10] [12:12:59] no [12:13:30] here goes [12:13:34] RECOVERY - check_ipsec on americium is OK: Strongswan OK - 1 ESP OK [12:13:35] PROBLEM - check_load on civi1001 is CRITICAL: CRITICAL - load average: 5.27, 72.63, 281.82 [12:13:46] paymentswiki doesn't talk to civi1001 directly, just to the db and queue [12:17:54] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=283 [critical = 10] [12:18:34] RECOVERY - check_load on civi1001 is OK: OK - load average: 0.75, 0.56, 0.23 [12:22:54] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=283 [critical = 10] [12:23:24] PROBLEM - check_ipsec on civi1001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: americium_v4 [12:23:34] PROBLEM - check_ipsec on americium is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: civi1001_v4 [12:27:10] hmmm [12:27:54] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=283 [critical = 10] [12:28:24] PROBLEM - check_ipsec on civi1001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: americium_v4 [12:28:34] PROBLEM - check_ipsec on americium is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: civi1001_v4 [12:30:16] i fixed ipsec [12:31:10] awesome [12:32:54] PROBLEM - check_rsyslog_backlog on frdb1001 is CRITICAL: CRITICAL frlog1001=282 [critical = 10] [12:33:24] RECOVERY - check_ipsec on civi1001 is OK: Strongswan OK - 1 ESP OK [12:33:34] RECOVERY - check_ipsec on americium is OK: Strongswan OK - 1 ESP OK [12:37:44] RECOVERY - check_rsyslog_backlog on frdb1001 is OK: OK [13:25:01] Jeff_Green: ugh [13:25:41] my ISP has been down for 2 days and my phone died from being left on as access point [13:31:07] well we should probably replace americium [14:35:11] PROBLEM - check_rsyslog_backlog on payments1004 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [14:39:01] PROBLEM - check_rsyslog_backlog on payments1004 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [14:42:31] PROBLEM - check_rsyslog_backlog on payments1004 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [14:44:21] weird yo [14:44:39] appears to have been stuck on the host [14:44:54] strong solar winds today [14:47:41] RECOVERY - check_rsyslog_backlog on payments1004 is OK: OK [14:48:26] https://s6.io/ [14:49:40] soundtrack for viewing that website: https://www.youtube.com/watch?v=IVJD3dL4diY [15:58:43] crud, those bad method / empty post errors seem to be getting worse [15:59:38] fr-tech let's figure out how to instrument those and get some good logging of where we lose the data [17:05:20] saurabhbatra: I have confirmed via calendar that nobody here is doing anything critical :p [17:06:09] hi fr-tech, awight and i were just talking about completing the fraud detection project by integrating the API and the model with the current transaction workflow [17:07:21] wondering what you guys think might be the best way to go about things [17:08:46] awight: i think the first goal was to integrate it but only log API results (fraud scores) in the db [17:09:41] and use those later to evaluate how the model performs on realtime data [17:09:43] That sounds great, from what I understand. [17:10:14] Remind me, did you end up using entirely features from CiviCRM, or do we need to capture ephemeral data from the initial payments requests? [17:11:09] looking here: https://github.com/saurabhbatra96/wmf-fd-api#request-definition [17:12:34] i think we'll have the data we need at the time of the transaction initiation [17:12:43] but before it goes into civi [17:13:08] i'm not sure i remember the flow correctly, but only successful transactions make it to Civi if i'm correct [17:14:25] Hi! [17:15:07] hello ejegg! [17:15:22] So, we have a sort-of pluggable way to add a fraud filter lookup to the DonationInterface process [17:15:55] We would create another class like this one: [17:15:57] https://github.com/wikimedia/mediawiki-extensions-DonationInterface/blob/master/extras/custom_filters/filters/minfraud/minfraud.body.php [17:16:06] i remember having this conversation in the past with you i think... [17:16:31] although it was about the geolocation api [17:16:32] ah yeah [17:17:09] so, this new class we'd write, would have access to the results of the geolocation [17:17:48] ejegg: Where would something like this sit? It's probably too complex for PCI, so I guess it would be a new node in the same subnet as Civi? [17:18:44] might be an over-generalization but that looks like a fancy api call wrapper [17:19:03] saurabhbatra: yep, the minfraud one is just that [17:19:28] so what if, say i wanted to use the minfraud score in my api call wrapper [17:19:50] * awight leans forward [17:19:52] awight: right, i figure it would have to be inside the boundary [17:20:27] call to our API has to sit at the end of the pipeline, because we're using a bunch of filter scores [17:20:46] saurabhbatra: hmm, I think the order of the fraud filters is defined by the config array that says which ones we want to run [17:21:18] ooo now i understand [17:21:21] sweet [17:21:30] and I think they share a common object that accumulates the scores [17:22:07] saurabhbatra: Running the scoring is nearly instantaneous, IIRC? Just making sure that we aren't adding latency. [17:22:26] https://github.com/wikimedia/mediawiki-extensions-DonationInterface/blob/master/extras/custom_filters/custom_filters.body.php [17:22:43] ^^^ is the thing that collects the scores from multiple filters [17:23:12] awight: correct, instantaneous [17:25:38] ejegg: Sorry if this is rhetorical but is there a staging environment for payments, where we can test the integration w/o disturbing production? [17:26:01] awight: argh, still no [17:27:09] I'm concerned that we might not be able to progress beyond local dev boxes until code thaw [17:27:33] https://github.com/wikimedia/mediawiki-extensions-DonationInterface/blob/master/extras/custom_filters/filters/minfraud/minfraud.body.php#L309 [17:28:06] this is how we access the filter scores? [17:28:36] awight: ah, good point [17:30:14] OTOH, maybe payment staging is a perfect "root cellar" project during the big fundraising drive :) [17:37:00] saurabhbatra: I think that function you found is actually fetching configuration, not results. [17:38:42] Seems that fraud score components are added to https://github.com/wikimedia/mediawiki-extensions-DonationInterface/blob/master/extras/custom_filters/custom_filters.body.php using addRiskScore, but I don't see any getters for retrieving the granular results. [17:40:51] awight: ah yes i see the results go into the cfo [17:40:53] Cross-filter dependencies is something we'd want fr-tech design help with, for sure... [17:40:57] could definitely add getters there [17:41:40] It seems reasonable, although also nice that the internal risk score components are opaque [17:43:47] ejegg: I see that cfo->validate has the side effect of calling gateway_adapter->setRiskScore [17:44:11] yeah, that's silly [17:44:28] gateway adapter should just ask for those after running the filters [17:44:48] kk, good. Cos it would be very fragile to rely on the side effet to call gateway_adapter->getRiskScore in another filter [17:44:52] (and provide all the settings up front) [17:47:57] i think i'm following [17:49:35] but if we don't have a staging, is there a way to run this locally? [17:49:49] (i'm new!) [17:50:40] Yes, it can definitely be run locally, but setting up a local payments frontend is not terribly fun. [17:50:57] I think mediawiki-vagrant can help... [17:52:03] yes, mw-vagrant's payment role is working pretty well these days [17:52:39] i guess have a first step then [17:52:50] *i have [17:53:02] whew! okay that's fantastic, looking at the systems it provisions you'll get DonationInterface and Civi [17:53:30] sorry to bail, but have to run out for ~30 mins [17:53:37] saurabhbatra: https://www.mediawiki.org/wiki/MediaWiki-Vagrant [17:53:44] saurabhbatra: vagrant roles enable payments [17:53:47] o/ [17:54:05] ejegg: thanks for the help! [17:54:20] awight: i'll bug you some more tomorrow or day after :-) [17:54:56] i'll set up vagrant and this virtualbox on my laptop and tinker around a bit [17:55:05] and virtualbox* [17:55:20] o/ [18:34:45] Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising-Backlog: Make sure audit parsers don't run at the same time as the disk array check - https://phabricator.wikimedia.org/T206571 (XenoRyet) a:XenoRyet [18:35:33] PROBLEM - check_rsyslog_backlog on pay-lvs1001 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [18:40:33] PROBLEM - check_rsyslog_backlog on pay-lvs1001 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [18:45:33] PROBLEM - check_rsyslog_backlog on pay-lvs1001 is CRITICAL: CRITICAL frlog1001=10 [critical = 10] [18:50:33] RECOVERY - check_rsyslog_backlog on pay-lvs1001 is OK: OK [19:27:24] fundraising-tech-ops: rsyslog stops reporting - https://phabricator.wikimedia.org/T207882 (cwdent) [19:32:32] XenoRyet: tech talk? [19:32:40] just on my way in. [19:32:54] headphones are being cranky [20:16:49] ah crap, all my DonationInterface tests are failing under PHP 7.3 [20:17:54] fr-tech anyone else on PHP 7.3? [20:22:02] (PS1) Ejegg: Fix continue statement breaking tests under PHP 7.3 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/469502 [20:23:51] (PS2) Ejegg: Fix continue statement breaking tests under PHP 7.3 [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/469502 [20:32:42] (PS1) Ejegg: PayPal IPNs: derive ct_id from order ID [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/469505 (https://phabricator.wikimedia.org/T202583) [20:32:44] (PS1) Ejegg: Debug logging for PayPal pending db lookup [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/469506 [20:52:17] (PS2) Ejegg: PayPal IPNs: derive ct_id from order ID [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/469505 (https://phabricator.wikimedia.org/T202583) [20:52:19] (PS2) Ejegg: Debug logging for PayPal pending db lookup [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/469506 [20:52:21] (PS1) Ejegg: Update PayPal EC test messages with sequence # oids [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/469513 (https://phabricator.wikimedia.org/T202583) [20:54:08] (PS1) Ejegg: Send order_id for all the PayPal merchant reference fields [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/469514 (https://phabricator.wikimedia.org/T202583) [20:58:40] (CR) Ejegg: [C: 2] Test data cleanup to better support multiple local runs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469157 (owner: Eileen) [21:04:30] (Merged) jenkins-bot: Test data cleanup to better support multiple local runs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469157 (owner: Eileen) [21:08:22] (CR) Ejegg: [C: 2] Fix extraneous links to 'view mailing report' from activities on activity tab. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/469138 (https://phabricator.wikimedia.org/T204908) (owner: Eileen) [21:14:06] (Merged) jenkins-bot: Fix extraneous links to 'view mailing report' from activities on activity tab. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/469138 (https://phabricator.wikimedia.org/T204908) (owner: Eileen) [21:23:27] fr-tech: do we know why donatewiki catchpoint is flapping? [21:27:24] cwd no idea [21:27:43] it's on the main cluster - does the wider ops team have any idea? [21:33:03] haven't checked [21:44:36] (PS3) Ejegg: Fix error on deleting activities with more than one link to the same contact. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469142 (https://phabricator.wikimedia.org/T204063) (owner: Eileen) [21:46:19] heading to dinner with family but will have laptop [21:46:31] (CR) Ejegg: [C: 2] "Looks like a good fix! One question about the return value." (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469142 (https://phabricator.wikimedia.org/T204063) (owner: Eileen) [21:46:42] (PS3) Ejegg: Alter permission for 'forgetme' from administer CiviCRM to edit all contacts [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469144 (https://phabricator.wikimedia.org/T204063) (owner: Eileen) [21:47:12] (CR) Ejegg: [C: 2] Alter permission for 'forgetme' from administer CiviCRM to edit all contacts [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469144 (https://phabricator.wikimedia.org/T204063) (owner: Eileen) [21:53:03] (Merged) jenkins-bot: Fix error on deleting activities with more than one link to the same contact. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469142 (https://phabricator.wikimedia.org/T204063) (owner: Eileen) [21:54:17] (Merged) jenkins-bot: Alter permission for 'forgetme' from administer CiviCRM to edit all contacts [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469144 (https://phabricator.wikimedia.org/T204063) (owner: Eileen) [22:01:04] (PS1) Ejegg: Ingenico: throw ex instead of making bad requests [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/469526 (https://phabricator.wikimedia.org/T202417) [22:01:24] (CR) Eileen: Fix error on deleting activities with more than one link to the same contact. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469142 (https://phabricator.wikimedia.org/T204063) (owner: Eileen) [22:04:20] ejegg: what was that UTC function you mentioned? [22:05:27] ah use SmashPig\Core\UtcDate; [22:06:42] yep yep! [22:06:45] (PS3) Eileen: Add a statement about server load to inform import timing decisions. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/468174 (https://phabricator.wikimedia.org/T204008) [22:07:10] ejegg: ok updated [22:07:14] thanks! [22:07:31] looks like they're shutting down here at the cafe, so I'mma relocate [22:07:55] does any one actually hear David’s voice in their head saying things like ‘start from the right of the sprint’ on a regular basis? [22:12:46] (PS1) Eileen: CiviCRM submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469528 [22:13:00] (CR) Eileen: [C: 2] CiviCRM submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469528 (owner: Eileen) [22:16:14] (Merged) jenkins-bot: CiviCRM submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/469528 (owner: Eileen) [22:19:26] what's the donatewiki catchpoint? [22:24:05] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/469530 [22:25:20] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/469530 (owner: Eileen) [22:26:02] ejegg|afk: I am doing a small deploy but vendor submodule seems like maybe it needs an update in deployment? [22:26:27] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/469530 (owner: Eileen) [22:27:34] !log civicrm revision changed from 1c0a1b2406 to 97506677e8, config revision is c0a8be03a1 [22:27:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:29:19] Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising-Backlog: Make sure audit parsers don't run at the same time as the disk array check - https://phabricator.wikimedia.org/T206571 (XenoRyet) Ingenico was the only one that conflicted, bumped it's schedule an hour later. [22:31:36] Fundraising Sprint Owls, Fundraising Sprint Pluto is still a planet, Fundraising Sprint Queue is pronounced GJif, Fundraising Sprint Raw data can give you salmonella, and 5 others: Import log files not including all skipped rows - https://phabricator.wikimedia.org/T200031 (Eileenmcnaughton) @Lean... [22:41:29] eileen_: ok, let me check on that [22:42:55] eileen_: crm/vendor looks good to me [22:43:40] composer install --no-dev says 'Nothing to install or update' [22:43:54] ejegg: ok I have local commit as 6782e3c42abf55aee278206286ed9 but submodule commit as 294c1d6ac64237cd684329f66 [22:44:03] on deployment branch [22:45:31] that 294... looks right [22:45:39] where did you r 678... come from? [22:45:51] ah, that was the previous one [22:46:15] if you pull master in vendor it should get you to 294... [22:48:59] ejegg: ah ok it was me then [23:07:31] (PS1) Ejegg: PayPal audit: get full order_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/469540 (https://phabricator.wikimedia.org/T202583) [23:08:52] (CR) jerkins-bot: [V: -1] PayPal audit: get full order_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/469540 (https://phabricator.wikimedia.org/T202583) (owner: Ejegg) [23:12:55] (PS2) Ejegg: PayPal audit: get full order_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/469540 (https://phabricator.wikimedia.org/T202583) [23:13:23] (CR) jerkins-bot: [V: -1] PayPal audit: get full order_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/469540 (https://phabricator.wikimedia.org/T202583) (owner: Ejegg) [23:16:50] (PS3) Ejegg: PayPal audit: get full order_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/469540 (https://phabricator.wikimedia.org/T202583) [23:17:46] (CR) jerkins-bot: [V: -1] PayPal audit: get full order_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/469540 (https://phabricator.wikimedia.org/T202583) (owner: Ejegg) [23:28:37] grr, why is it failing pyflakes? \. is totally legit for escaping in a regex, right? [23:30:49] (PS4) Ejegg: PayPal audit: get full order_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/469540 (https://phabricator.wikimedia.org/T202583) [23:31:26] (CR) jerkins-bot: [V: -1] PayPal audit: get full order_id [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/469540 (https://phabricator.wikimedia.org/T202583) (owner: Ejegg) [23:33:41] (CR) Eileen: [C: 2] "OK - I checked & phpdocs say break is the same as continue in this context so I'm happy this is not a behaviour change" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/469502 (owner: Ejegg)