[00:22:44] #988: (AW) O:AW|TS:B|P:MH|T:TD -- https://mingle.corp.wikimedia.org/projects/fundraiser_2012/cards/988 [00:24:44] #986: (AW) Description changed -- https://mingle.corp.wikimedia.org/projects/fundraiser_2012/cards/986 [00:50:04] marktraceur: does DI even have unit tests enabled on merge? [15:58:58] * marktraceur waves from home [15:59:10] I was coughing up both lungs yesterday but I haven't any meetings today, so home [15:59:29] * K4-713 waves at marktraceur [15:59:40] That sounds unpleasant [15:59:47] It is! [15:59:51] :( [15:59:56] I'll be all right [16:00:07] At least I'm not writhing and trying to sleep still. [16:00:15] Well, you know: Fluids. [16:00:26] *nod* [16:00:26] I *had* to suggest something. [16:00:31] :p [16:00:51] I'll get some oobleck or summat [16:00:56] Hey Jeff_Green: Can we talk Maintenance Day? [16:01:31] marktraceur: Whiskey is a fluid. Just sayin'. [16:01:45] K4-713: sure [16:02:54] K4-713: Aye but I think YuviPanda would be mad at me if I got drunk at 09:00 and rejected his patch with a simple "I'm sorry, this code was written by a doody-head" [16:03:15] This is only a hypothesis. [16:03:23] bahaha [16:03:28] Jeff_Green: So, next week? [16:04:03] K4-713: I can do mon-weds [16:04:07] thursday is jury duty [16:04:11] Unless you really want to go tomorrow, they're eyeballing the slow dropoff in donations, and it hasn't evened out yet. So, they want more time for the eyeballing. [16:04:13] friday is unknown [16:04:26] K4-713: next week is good [16:04:51] Cool. How about we say next Wednesday? [16:05:29] how about tuesday? that gives us a day to discover/unbreak sneaky things before jury duty [16:05:53] Yeah, good idea. [16:06:04] I'll get Tuesday, then. [16:06:08] ok cool [16:06:37] Any idea what time you'll be starting in? I want to make sure they account for the long tail. [16:07:12] on cutover? i guess that depends how closely you think you guys want to watch things [16:07:37] I can now do it as a DNS flip but there will be a window of slop [16:07:52] Sure, makes sense. [16:07:54] so I guess I should write-lock db1008 at the same time to prevent accidental writes there [16:09:38] I guess I'd like to be online and monitoring when you start... so I'm probably going to suggest 10am PST, but can be talked in to just about anything. [16:09:56] 10A sounds good [16:10:29] since we have 3 db's I see the cutover as going sorta like: [16:10:47] - stop writes as best as possible by crazy magic [16:11:00] - write-lock db1008 [16:11:09] - change master to db1025 and test replication [16:11:12] - switch DNS alias [16:11:21] - switch on writes by undoing crazy magic [16:11:34] and then I can work on db1008 without disrupting fundraising foo [16:11:50] We... do have a maintenance mode for payments. But, last time we tried it, we only had limited success. I should try to dust that thing off and figure out what didn't quite work before. [16:12:55] k [16:13:19] So, my bit would be put payments and civi into maintenance mode. Trying to think of anything else that would write to 1008... [16:13:42] stats-related cron/jenkins jobs i guess [16:13:55] Hm. Stats. [16:14:05] mwalker: Hi. [16:14:11] let's start a wiki cutover doc [16:14:21] morning K4-713 [16:14:27] the-wub: *waves* [16:14:37] Jeff_Green: sounds good. [16:14:45] Should be pretty prepared by Tuesday. :) [16:14:52] hi mwalker. want to talk about emails soon? [16:15:08] Hey, I got him first. :p [16:15:10] K4-713: officewiki you think? [16:15:16] Nah, collab. [16:15:49] wooo I have auth there, that's pretty exciting [16:15:59] There's a long-winded explanation of where everything lives now, in the Fundraising Engineering category on... ah, all the wikis. Aside from tech, where it's just the Fundraising category. [16:16:41] the-wub: yep yep -- I checked the basic SQL script into the wikimedia/fundraising/tools git repo [16:17:21] so grab that and place it somewhere useful to you [16:17:45] Jeff_Green: Basically, collab is the preferred location for data in that weird sensitive category where we don't want to air it out to the world, but it's still okay to be on a wiki at all, and wikitech is the preferred place where all the publicly interesting and not-sensitive or us-centric stuff goes. [16:17:53] the-wub: either via git or https://git.wikimedia.org/raw/wikimedia%2Ffundraising%2Ftools/c07483fdf46637ffe2b87e4e0146778ff6f2da78/silverpop_export.sql directly [16:18:15] K4-713: cool [16:28:10] Jeff_Green: do you know why the-wub would be unable to create indexes in the mwalker database? [16:28:34] there's a distinct mysql priv for that, which I probably didn't grant him [16:28:36] looking [16:28:42] yep... "ERROR 1142 (42000): INDEX command denied to user 'pcoombe'@'208.80.154.6' for table 'silverpop_export'" [16:29:05] yes [16:29:21] mysql privs are granular in odd ways [16:31:58] try now [16:32:53] the-wub: ^ [16:33:12] it works now. thanks Jeff_Green :) [16:33:19] the-wub: yw [16:35:31] K4-713: you were wanting to bug me about something? [16:35:46] Aye. [16:36:05] We've got a maintenance day next Tuesday, in which Jeff is going to switch out the db master. [16:36:43] We need to know what (if anything) we need to do other than put payments and civi in maintenance mode, to prevent attempted db writes during the cutover. [16:36:49] Mostly we were wondering about stats. [16:37:37] Oh, and: Do the listeners do any direct db manipulation, or is it all queue-based? [16:37:50] stats only reads right now [16:38:01] and we can just disable the job in jenkins [16:38:10] :) [16:38:14] the listeners are all queue based [16:38:45] This conversation makes me happy. [16:38:56] Is there anything else we need to examine? [16:39:00] https://collab.wikimedia.org/wiki/Fundraising/Engineering/Notes/201307-fundraisingdb-master-cutover [16:39:06] pls go crazy marking that up [16:39:25] And, yeah: Like I said, it's next Tuesday. So we've got some time. [16:40:19] hurm... peter's banner scripts I guess [16:40:25] not sure where those are configured [16:41:13] Hmmmmmm. [16:48:29] mwalker: Yeah, we should track that down before Tuesday, probably. [16:48:44] yep; wandering around now trying to find it [16:48:54] Not sure where they are, or where they store their db creds... thanks. :) [16:56:10] Jeff_Green: :'( pcoombe apparently does not have select on geodata -- can you give him that? [16:56:23] hrmf. i thought everybody did... [16:56:25] fixing. [16:56:26] *geonames [17:00:00] jvandavier: you wfh today? [17:00:33] negative, will be in soon - something happened to the MUNI lines earlier, so trains stopped for a bit [17:18:56] Jeff_Green: any luck? still not working for me [17:19:12] oh shite. i got sidetracked and totally spaced. sorry. fixing for real now [17:21:08] huh. try again pls? [17:22:23] nope, still same problem [17:22:30] ERROR 1142 (42000): SELECT command denied to user 'pcoombe'@'208.80.154.6' for table 'geonames' [17:22:56] what's the query? [17:25:08] http://pastebin.com/uHrwUSGK [17:25:35] strange [17:25:39] so 8.1 completed? [17:25:44] but 8.2 is failing? [17:26:54] the-wub: try again. apparently wildcarding both user+host doesn't work as expected so I granted your user specifically [17:27:15] i haven't managed to find documentation for proper syntax [17:27:34] oh sorry mwalker, it was 8.1. pasted the wrong thing [17:27:47] ah; gotcha [17:27:51] but it's working now! thanks again Jeff_Green [17:27:57] ok [17:28:12] mwalker: I think I may finally know what was going wrong last week with db1025 vs your queries from hell [17:28:21] oh? [17:28:38] I suspect your tables were being created as myisam b/c that's still the default table type for db1008 [17:28:43] and db1025 is totally de-tuned for myisam [17:30:33] ah... that would make sense... I'm not specifying an engine in these queries [17:41:36] ok -- I'm going to put on pants and walk in -- be there in 20 [18:15:15] Build failure? [18:15:50] niiiice failure [18:15:58] dafuq... [18:16:22] So, I'm getting "Civi core went crazy". [18:16:27] it could have been a timeout [18:16:37] the-wub: and I are currently pulling from civicrm_contact [18:16:44] (or I am anyways) [18:16:51] oho [18:17:36] ya; totally it: Lock wait timeout exceeded; try restarting transaction") [18:18:42] ok; we're out of the table [18:18:45] should be back to nomrla [18:19:10] Just went back to normal, yes. [18:19:19] ... [18:19:27] What db were you on? [18:19:44] 1008 [18:19:57] Can you use 1025? :) [18:19:58] because we're pulling data into mwalker for export [18:20:04] sadly no :( it's not writeable [18:20:10] Ah, I See. [18:20:29] Well... shall I turn off qc while you're in there, then? [18:20:45] I have started the 5 minute login process. [18:21:03] * K4-713 makes rude gestures at Jenkins [18:22:35] nah; it's cool [18:22:46] that was the last long running query we needed to make off of a live DB [18:23:05] It finished completely? [18:23:21] from civicrm_contacts [18:23:26] Or, to put it another way, you didn't kill it? [18:23:29] nope [18:23:36] Okay, never mind, then. :) [20:05:31] http://www.theatlanticwire.com/entertainment/2013/07/giant-statue-wet-colin-firth-obviously-kind-creepy/66918/ [20:06:56] Oh hai morebots. [20:10:38] oho. I was wondering where he went. [20:10:47] But, ah... idly. [20:11:01] Not with intent to actually find out or do anything about it. [22:15:33] !log updating payments.wm.o from eee8689ab6466d51f9991c90f73004ad6c9d0b5b to 1c360e2cde650d1195f04aee294ddd3e0f5f9052 [22:15:42] Logged the message, Master [22:17:08] *twiddles thumbs for a donation or two* [22:18:12] excellent [22:18:15] looks like it worked [22:47:47] K4-713: can I get you to review https://gerrit.wikimedia.org/r/72855 [22:47:58] it's a patch to contriubtion reporting for this years stats [22:57:22] mwalker: Contribution reporting isn't completely broken? [22:57:30] suprisingly no [22:57:35] it still is chugging along [22:57:46] ...huh. Well, okay then. [23:30:05] ah... [23:30:16] you know how you asked how contributionreporting was actually working? [23:30:21] turns out it isn't [23:30:25] it was pulling from db1025 [23:30:34] which is now behind the frack firewall [23:30:35] Ah! The world realigns with my perceptions. [23:30:51] * K4-713 nods, looks slightly less confused. [23:31:43] soooo [23:31:50] thoughts on statistics?