[00:11:53] (PS1) Ejegg: Add host name to failmail subject [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346482 (https://phabricator.wikimedia.org/T162214) [00:12:20] (CR) jerkins-bot: [V: -1] Add host name to failmail subject [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346482 (https://phabricator.wikimedia.org/T162214) (owner: Ejegg) [00:21:14] (PS2) Eileen: Fix Benevity to update existing contact with email. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/345268 (https://phabricator.wikimedia.org/T161666) [00:22:03] (CR) Eileen: "I've changed it to call only the email update when street_address is null. We do have code 'protecting' against empty changes but there is" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/345268 (https://phabricator.wikimedia.org/T161666) (owner: Eileen) [00:45:46] (PS2) Awight: Add host name to failmail subject [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346482 (https://phabricator.wikimedia.org/T162214) (owner: Ejegg) [00:45:57] (CR) Awight: [C: 2] "That's an improvement!" [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346482 (https://phabricator.wikimedia.org/T162214) (owner: Ejegg) [00:46:25] (Merged) jenkins-bot: Add host name to failmail subject [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346482 (https://phabricator.wikimedia.org/T162214) (owner: Ejegg) [10:15:00] Wikimedia-Fundraising-Campaigns (FY Q3 Jan-Mar 2017), Wikimedia-CentralNotice-Administration (Q1-2017): WMF Fundraising - 2016-17 - Austria (Mar 7 - Apr 4) - https://phabricator.wikimedia.org/T157119#3156817 (Pcoombe) Open>Resolved a:Pcoombe [14:55:43] fr-tech if anyone wants to shut up the smashpig iDEAL lookup failmail, there's this patch (and its parent) to review: https://gerrit.wikimedia.org/r/345790 [16:44:30] Wikimedia-Fundraising-Campaigns (Wikimedia-Fundraising-Campaigns (FY Q4 Apr-Jun 2017)), Wikimedia-CentralNotice-Administration (Q2-2017): WMF Fundraising - 2016-17 - Latin America 1 (May 2 - May 30) - https://phabricator.wikimedia.org/T162287#3157901 (Pcoombe) [16:45:53] Wikimedia-Fundraising: Fundraising banner pre-test in Latin America - https://phabricator.wikimedia.org/T162288#3157915 (Pcoombe) [16:46:06] Wikimedia-Fundraising: Fundraising banner pre-test in Latin America - https://phabricator.wikimedia.org/T162288#3157915 (Pcoombe) a:Pcoombe [16:54:43] (PS1) Awight: Don't log logging settings [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346571 [17:00:19] fr-tech: Why I Can't Go Out With You: [17:00:19] I'd LOVE to, but... [17:00:19] -- I've got a Friends of the Lowly Rutabaga meeting. [17:00:19] -- I promised to help a friend fold road maps. [17:00:19] -- I've been scheduled for a karma transplant. [17:00:19] -- I'm staying home to work on my cottage cheese sculpture. [17:00:19] -- It's my parakeet's bowling night. [17:00:20] -- I'm building a plant from a kit. [17:00:20] -- There's a disturbance in the Force. [17:00:21] -- I'm doing door-to-door collecting for static cling. [17:00:21] -- I'm teaching my ferret to yodel. [17:00:22] -- My crayons all melted together. [17:04:14] slander: you wish. [17:06:08] awight: does the timeout actually kill the subprocess? [17:06:45] It looks like it did, but I'm not sure how [17:06:52] i deployed an ejegg change last night to raise the timeout for the recurring job [17:07:01] Heheh ferret yodeling [17:07:08] My ferrety-favorite kind [17:07:17] that export is huge so we should probably raise the timeout significantly [17:07:45] I'm not sure we should rush straight for increasing the timeouts [17:08:02] e.g. Fundraiser Public Data has taken exactly 2 minutes 50 seconds forever [17:08:09] If it's taking longer, then something's wrong [17:08:16] well sure [17:08:20] but the silverpop export [17:08:25] doesn't that alwasy take hours? [17:08:30] oh that [17:08:39] yeah totally needs more timeout leash [17:08:44] right thx [17:08:48] that timeout should be 0 [17:08:49] sorry yeah not sure about the data export [17:08:59] and we leave it to the substeps to time out [17:09:54] (CR) Ejegg: [C: 2] Fix Benevity to update existing contact with email. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/345268 (https://phabricator.wikimedia.org/T161666) (owner: Eileen) [17:10:06] what is urgent right now? anything more so than moving the server? [17:10:38] Pull out the stops! [17:10:38] it might be nice to save the p-c hacking for the OS it will be on long term [17:10:59] I'm happy to leave it semi-stable until we're migrated [17:11:04] cool [17:11:05] ... I think [17:11:17] i don't *see* anything on serious fire [17:11:43] (PS1) Awight: Split job spec from job runner [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346574 [17:11:51] Nevermind the massive changes ;) ^ [17:11:52] awight is the code deployed to treat zero as no timeout? [17:12:05] hum lemme see. It's the wmf-1.0.4 tag. [17:12:19] the deb version is .4 [17:12:20] (PS2) Awight: Split job spec from job runner [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346574 [17:12:39] ejegg: yes [17:12:40] if we push to move the box now i can stop building for 2 OSes :) [17:12:56] cwd: I would love that. I'll even go python3 just to be a jerk [17:13:00] awight: oh cool, I'll make a puppet patch to remove that default [17:13:14] we would have asyncio available then too [17:13:21] ejegg: wait, what about just specifying timeout=0 where we want that [17:13:27] I like the idea of having a reasonable default [17:13:36] yeah? [17:13:52] Without that, our most mundane jobs become a big liability [17:13:53] There are a lot of jobs that time themselves, I guess specify 0 for all of them [17:14:11] no I would specify 2x maximum, dontchu think? [17:14:24] (Merged) jenkins-bot: Fix Benevity to update existing contact with email. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/345268 (https://phabricator.wikimedia.org/T161666) (owner: Eileen) [17:14:24] in case their own timing mechanisms break? [17:14:28] yep [17:14:30] cos PHP [17:14:40] And we do set_timeout magic [17:14:48] hmm [17:15:05] I can be swayed, try me :) [17:15:06] i don't like specifying timeouts in two places [17:15:20] We already have that, but it's hidden--the PHP interpreter defaults to 10 minutes [17:15:24] and we never had a timeout for most jobs under jenkins [17:15:33] oh yeah, that thing [17:15:59] we should get a definite answer about how it managed to kill the subprocess [17:16:14] i pretty much thought that would keep rolling even if python died but maybe not? [17:16:23] cwd how jenkins killed it? [17:16:35] the public data export that failed this morning, for example... it was supposed to take 2.9 minutes, but got stuck in some kind of infinite loop or deadlocked query. I'm glad that it timed out [17:16:39] pretty sure it was sending sigterm [17:16:47] ejegg: nah the public data export this morning [17:16:53] ah, k [17:16:59] weird. [17:17:05] ejegg: what's weird is that the parent "public data export" timed out, but it looks like the child died too [17:17:31] btw, the parent/child job thing makes me a little sad. Feel free to rethink that [17:17:38] e.g. astropay_audit [17:18:18] So we should urgently rerun the SP export, eh? [17:18:28] I should have done that last night but got scared and lazy. [17:18:32] yeah, seems so [17:18:41] with a corrected timeout [17:18:46] kk will do [17:19:30] * awight makes a short hop into the air at seeing your source-controlled, commented changed to the job config [17:19:43] \o/ [17:19:56] Yays! [17:20:21] yes I'll grammard that latter [17:20:34] ladle rat rotten hut [17:21:23] Wikimedia-Fundraising: Desktop small nag appears too early on Italian Wikipedia - https://phabricator.wikimedia.org/T162289#3158006 (Pcoombe) [17:22:21] ejegg: We should find consensus on the timeout issue... Think it's worth discussing in a task, or dragging in more team? [17:23:06] ejegg: i did comment on that GC fix awhile ago, it would be nice to be able to use the canonical list of status codes vs the magic number... [17:23:28] cwd ooh, sorry [17:23:38] I'll revisit that one [17:25:25] anyone have news for scrum of scrums? [17:25:29] fr-tech ^^^ [17:25:46] None here [17:26:23] You could brag about slaying Jenkins, if it seems appropriate... [17:26:30] yepyep! [17:26:32] I would *not* mind having other groups at WMF using this tool [17:28:10] ejegg: nothing here for now, thanks! [17:28:37] ejegg: Did you kick off a SP export? [17:28:51] oh that must be a stale lockfile :( [17:29:42] d'oh! [17:29:51] ccogdill: I just kicked the SP export, so if all goes well it'll be ready in about 5 hours. Sorry I didn't get to it last night! [17:30:09] oh cool, thank yoU! [17:30:13] ejegg: hmm, "writing lockfile" powered right through w/o complaining about a stale lock :-/ [17:30:14] that’ll do the trick [17:30:17] ccogdill: of curse [17:31:02] well that feels ominous :P [17:31:23] whoohahaha [17:54:32] (CR) Ejegg: [C: 2] Don't log logging settings [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346571 (owner: Awight) [17:56:08] (CR) Ejegg: [WIP] Custom campaign mixin param handlers (2 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/343953 (https://phabricator.wikimedia.org/T144453) (owner: AndyRussG) [18:01:24] (Merged) jenkins-bot: Don't log logging settings [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346571 (owner: Awight) [18:02:04] !log rerunning paypal_audit [18:02:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:16:41] (PS1) Awight: Include logfile path in failmails [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346583 [18:18:08] Oooh comments on the WIP :) [18:18:37] (PS1) Awight: Strip extra newline after job status [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346585 [18:18:44] AndyRussG: just trivia [18:19:02] ejegg: thanks for starting to dig! [18:19:23] my pleasure! [18:19:32] I have a heavily modified local version of the follow-on, was gonna upload when it's doing stuff again :) [18:19:36] nice to see the admin ui js cleanup patches too [18:20:17] ejegg: heh yeah there was quite a bit of crusty spillage at the bottom of those cuboards, eh [18:20:36] cupboards [18:21:37] Fundraising-Backlog: process-control should tail the job logs if run interactively - https://phabricator.wikimedia.org/T162294#3158270 (awight) [18:27:47] (PS1) Awight: Mirror child process output to console [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346586 (https://phabricator.wikimedia.org/T162294) [18:29:57] (PS3) Awight: Split job spec from job runner [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346574 [18:29:59] (PS2) Awight: Include logfile path in failmails [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346583 [18:30:01] (PS2) Awight: Strip extra newline after job status [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346585 [18:30:03] (PS2) Awight: Mirror child process output to console [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346586 (https://phabricator.wikimedia.org/T162294) [18:37:16] (PS3) Awight: Mirror child process output to console [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346586 (https://phabricator.wikimedia.org/T162294) [18:50:36] fr-tech I'm heading back to the US, wish me luck! Back on in 4 hrs-ish from ATL [18:51:18] ejegg: have a good flight! [18:51:25] bittersweet! [18:51:28] (PS3) Ejegg: Admin UI: delay setup until document ready in campaignManager.js [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/345873 (https://phabricator.wikimedia.org/T144453) (owner: AndyRussG) [18:51:46] (CR) Ejegg: [C: 2] Admin UI: delay setup until document ready in campaignManager.js (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/345873 (https://phabricator.wikimedia.org/T144453) (owner: AndyRussG) [19:02:18] (CR) jerkins-bot: [V: -1] Mirror child process output to console [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346586 (https://phabricator.wikimedia.org/T162294) (owner: Awight) [19:03:49] (CR) Awight: "recheck" [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346586 (https://phabricator.wikimedia.org/T162294) (owner: Awight) [19:04:32] (Merged) jenkins-bot: Admin UI: delay setup until document ready in campaignManager.js [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/345873 (https://phabricator.wikimedia.org/T144453) (owner: AndyRussG) [19:09:10] (CR) Awight: [C: -1] "Though I'd rather fix the race condition." [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/345781 (owner: Awight) [19:10:27] Wikimedia-Fundraising: Fundraising banner pre-test in Latin America - https://phabricator.wikimedia.org/T162288#3158450 (Pcoombe) Tentatively scheduled for Thursday 6 April, 1600 UTC. [19:32:56] problem w / https://donate.wikimedia.org/wiki/Special:FundraiserLandingPage?uselang=en&country=NL will put in Phab but very ugly awight cwd [19:33:11] ok looking [19:33:21] oh god [19:33:44] MBeat: How many people have reported that? [19:33:54] three in last minute [19:34:01] Taking campaigns down... [19:34:04] the-wub: ^ [19:34:34] Fundraising-Backlog: donation page format error - https://phabricator.wikimedia.org/T162300#3158511 (MBeat33) [19:34:51] Fundraising-Backlog: donation page format error - https://phabricator.wikimedia.org/T162300#3158523 (MBeat33) p:Triage>Unbreak! [19:35:05] awwww [19:35:51] Hey all just saw the phab task [19:36:01] what the shit [19:36:11] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Unplanned-Sprint-Work: donation page format error - https://phabricator.wikimedia.org/T162300#3158541 (DStrine) [19:36:20] added to sprint: T162300 [19:36:21] T162300: donation page format error - https://phabricator.wikimedia.org/T162300 [19:37:18] !log disabled NL campaigns per T162300 [19:37:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:37:33] I'll email fr-online [19:37:45] MBeat: Campaigns are down, let us know if the complaints peter out [19:37:52] thanks awight and all [19:38:37] * AndyRussG poders the status of recentl-sent email is [19:39:09] RainbowSprinkles just reverted the train deploy on donatewiki [19:39:14] https://donate.wikimedia.org/wiki/Special:FundraiserLandingPage?uselang=en&country=NL&1 looks fine now [19:39:39] https://phabricator.wikimedia.org/T156184 [19:39:54] legoktm: thanks! [19:40:03] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Patch-For-Review, Unplanned-Sprint-Work: donation page format error - https://phabricator.wikimedia.org/T162300#3158511 (Legoktm) This was caused by {T156184}. [19:41:44] the FundraiserLandingPage thing shouldn't use in system messages [19:42:38] cwd: awight ejegg|away XenoRyet|afk MBeat Codgill says 60k emails went out this morning. I'm guessing they are going to see a broken form if they click [19:42:50] uh [19:42:55] dstrine: it should be fixed now? [19:42:56] OK, I'll reenable campaigns [19:43:05] legoktm: +1 it looks fixed, thanks for relaying the news! [19:43:10] ok :) [19:43:53] !log reenabled NL Fundraising campaigns [19:44:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:44:06] was this a specific form or all? what was the cause? [19:44:56] No idea yet. [19:45:06] ok [19:45:12] dstrine: donatewiki is on the main cluster and outside fr-tech's purview for the most part [19:45:37] so did an ops thing cause this? [19:45:39] * awight hangs out alone [19:46:03] dstrine: yeah, something about the MediaWiki train deployment [19:51:26] https://github.com/wikimedia/wikimedia-fundraising-process-control looks neat. It could use a more prominent license though. GPLv2 seems to only be documented in the debian control files. [19:51:38] bd808: oh hey, good call. Thanks! [19:51:47] You wanna use it, it's gonna cost you :p [19:52:00] * bd808 gave at the office [19:52:08] blood? [19:52:18] first born child [19:52:23] sounds bratty [19:53:21] bd808: :) [19:53:39] we finally killed a 6 year old jenkins instance that was the "job runner" [19:53:59] If Faidon sees it he will give you a lecture about putting your debian directory in the master branch too, but that's not the sort of thing I whine about [19:54:23] jenkins as a job management system is ... good call replacing it :) [19:54:34] uh huh... [19:55:00] it had grown bark and spines [19:58:27] jenkins can be nice for making job trees, but it always feels like the wrong tool for anything other than CI pipelines [19:59:25] yes [20:00:33] we used cron instead [20:00:37] as it was written [20:02:05] * Jeff_Green is happy we didn't end up using nodejs somehow :-P [20:08:14] Jeff_Green: I heard from cwd that we could do this civi maintenance as early as tomorrow. the best time for fr-online would probably be after 2pm but we need to confirm that. Is that a good time? shall we discuss other times? [20:08:32] awight: says he won't be there tomorrow but might not need to be. [20:08:44] not sure if ejegg|away can help [20:08:57] Yea I need to cut out at 5pm sharp tomorrow, but can attend the rest of the party. [20:10:00] dstrine: tomorrow is a bit dodgy for me because I have a hard stop at 6:30PM [20:10:11] i could do friday [20:10:16] could do weekend [20:10:28] lol friday or weekend?!?!? [20:10:31] EOL is sat right? [20:10:39] would be prudent to do it before then if possible [20:10:39] well there is that, it's true [20:11:05] we might be able to do this earlier tomorrow I'm checking in parallel with MG before I even send an email [20:11:11] it's not as though it turns into a pumpkin at midnight on friday, they just stop releasing updates [20:11:32] hehe [20:11:42] yeah not the end of the world [20:11:58] I suppose we could hack it crazy style by swapping IPs even if netops can't do it [20:12:16] hmmm that's interesting [20:12:17] it's just a lot of crumbs to pick up afterward that way [20:12:30] what kind of crumbs do you foresee? [20:13:52] DNS, stored SSH public keys, a lot of cleanup of inline comments in puppet... [20:14:22] there may be other things, I'll ponder what would have to happen to make it work that way [20:14:41] ah yeah [20:14:45] host entries would cry [20:14:55] that's what we would have done in the event of a hardware failure though, it's feasible [20:14:57] yeah [20:15:08] but for barium that's not so bad [20:15:17] nah luckily it's just internal stuff [20:15:20] for lutetium it would be more of a mess because of all the additional users [20:15:31] yep [20:15:43] dash... [20:15:47] I gotta figure out dash [20:15:54] is anyone using it right now? [20:16:02] cwd: no [20:16:12] oh, that's great then [20:16:34] cwd what were you saying about style sheets before? [20:17:33] Jeff_Green: pretty sure it's just the port i was using, i can verify one sec [20:17:45] k [20:17:48] i think civi generated some urls server side and so it was requesting a url from barium that didn't exist [20:18:01] cause it was generated on 1001 [20:18:08] weird [20:18:13] but the stylesheet link pointed to barium [20:18:19] anyway i'll verify [20:18:21] by hostname? [20:18:24] yeah [20:18:29] that's appalling [20:18:31] i'll map it to the same port and see [20:18:41] how does that not cause an SSL error? [20:19:47] i used /etc/hosts [20:20:05] but browsed to :highport [20:20:17] but it didn't generate the stylesheet link with the port [20:20:24] ahh, so if you actually hit that URL on civicrm.wm.o you get different content [20:21:14] that's what i'm guessing [20:21:17] figuring it out now [20:21:22] su for low ports and all [20:21:46] k [20:22:23] do you think Eileen will need to be on during the civi upgrade? does that dictate timing? [20:23:20] It wouldn't hurt for her to be available in case of surprises, but I don't know that she has to be there [20:25:02] the more I look at the IP swap approach the less it scares me [20:25:36] well this is weird [20:25:47] with the right url it's all 200s, but it still doesn't look right [20:26:30] it is missing some styles [21:03:01] cwd: Jeff_Green MG are all traveling later tomorrow. We could do this as early as noon pacific if that helps anyone [21:03:44] i can do that as long as we have a sane rollback plan [21:03:45] but 2pm pacific would be best so engage can still work [21:04:07] you guys tell me [21:04:21] errr [21:04:41] i have to be done by 6:15 eastern, because my son is in a play [21:04:49] the friday/weekend suggestion sounded problematic for work/life balnce [21:04:52] so earlier would be better [21:05:02] ah yeah [21:05:03] i don't mind timeshifting personally [21:06:46] Jeff_Green: I think rollback is simplified by not moving the databases. The only change that oldbox will see is that we're disabling the cronjobs, right? [21:06:57] Ah, unless you have to do IP wizardry [21:07:26] i think that's the easiest, scheduling with netops is going to get complicated [21:07:35] so we'd just swap IPs [21:07:47] so rollback is hopefully just swapping back? [21:08:34] yeah, unless there's some scenario where temporarily running the newer box could cause a data catastrophe? [21:08:35] & reenabling jobs [21:08:49] The only awkwardness I know if is that logs are starting to be fragmented [21:09:04] Some are in Jenkins, some in oldbox:/var/log/p-c, and some will be in newbox:/var/log/p-c [21:09:28] But other than logs, we don't write any important data to disk [21:09:38] we could merge the p-c stuff together [21:09:42] hum, maybe donations import rejected files. [21:09:44] cool. [21:09:57] yeah, names will be unique and directory structure identical. [21:10:12] the central logged stuff will have hostnames on it, but otherwise end up in the same place [21:10:51] is earlier in the day tomorrow not an option? [21:11:16] Fine for me, but dstrine was saying that Major Gifts is all traveling in the afternoon. [21:11:21] He said, 12pm Pacific is an option. [21:11:47] 12 is better from my perspective [21:11:51] fr-online wants to test astropay tomorrow morning from 9 and possibly to noon [21:11:59] will that interfere? [21:12:06] it seems like it should take an hour, which means it will take 3, which is just ok if we start at 12 [21:12:23] dstrine: We need to check with ccogdill, and not schedule emails. [21:13:06] awight she wanted to send some at 1pm tomorrow but says she doesn't HAVE to [21:13:11] kk ty [21:13:48] (PS1) Awight: [WIP] Fix double-failmail after timeout. [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346640 [21:14:28] meanwhile I gotta head to dinner now, I'll be back online later tonight to finish recloning frdb1002 [21:14:51] so shall I propose a noon to 3 pacific mainenance time? [21:14:55] *maintenance [21:14:55] (PS1) Awight: add license file [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346641 [21:15:12] dstrine: works for me [21:15:32] Anyone else prefer noon until Fri start of work? [21:15:42] We can always end early. [21:17:07] i'm good with that too [21:18:11] I will send an email to double check with fr-all [21:19:00] thanks dstrine! [21:19:06] * Jeff_Green goes for real now [21:20:21] (PS1) Awight: Remove debian metadata from the main repo [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346643 [21:24:25] I'm gonna commute like an abelian group operation [21:24:29] be back soon. [21:26:46] ok sent [22:14:20] cwd: has anything changed on civi within the last few minutes? Leanne just noticed the layout is all wonky [22:14:28] maybe something with the css? [22:14:29] https://www.screencast.com/t/qMD1EyEcuAP [22:14:35] looks like this^^ [22:14:50] wtf [22:15:01] Leanne says this just happened [22:15:08] dstrine: i regenerated the styles on the new server, i guess it must have affected the current one. sorry! [22:15:12] eileen: ^ anyidea? [22:17:03] totally confused about why that happened, i can fix it in a minute [22:17:34] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Unplanned-Sprint-Work: CSS messed up on Civi production - https://phabricator.wikimedia.org/T162317#3159014 (DStrine) [22:17:52] I made a task jsut in case [22:17:55] *just [22:18:03] cwd - it will be the business of cross-posting to that server, due to the new one pointing to prod [22:18:40] eileen: i'm confused, cross posting? [22:19:31] just changed it back [22:19:42] eileen: thank you! [22:19:52] every time I clicked submit on the new site I was winding up on prod [22:20:05] so what was being submitted where is a bit of an open question [22:20:28] i'll go ahead and leave that alone until we make the transition [22:21:10] this is the day of quick fixes [22:21:13] :) [22:25:26] * cwd wipes brow [22:25:38] i clearly don't understand civi well enough [22:25:49] i don't know how that affected prod [22:27:57] cwd it's a drupal thing, not a civi thing [22:28:25] err yeah sorry [22:28:32] is it in the db somehow? [22:28:41] did i point to a colorscheme that didn't exist on prod? [22:28:52] so the reason it's redirecting is the $base_url in settings.php [22:29:23] was it making a server side request that was unaffected by my hosts entry? [22:29:28] I think it's something like submit & it does part & then it loads (after the redirect) & does the rest [22:29:41] might have been mine not yours [22:29:55] ooh si si [22:47:16] (PS2) Awight: Remove debian metadata from the main repo [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346643 [22:50:21] awight: anything more we should do to prep? [22:51:32] cwd: Is there a list somewhere of steps we're taking, and known problems that we'll have to resolve? [22:52:24] let's start one [22:56:04] awight: https://etherpad.wikimedia.org/p/civi_transition [22:56:41] oh thanks! [23:03:06] (CR) Cdentinger: [C: 2] Remove debian metadata from the main repo [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346643 (owner: Awight) [23:03:49] (Merged) jenkins-bot: Remove debian metadata from the main repo [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346643 (owner: Awight) [23:16:19] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising-Backlog, Patch-For-Review, Unplanned-Sprint-Work: donation page format error - https://phabricator.wikimedia.org/T162300#3159215 (MBeat33) Open>Resolved a:MBeat33 After the rollback the form is working and we've seen no... [23:30:50] (PS1) Awight: Document a bit [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346664 [23:38:49] (PS2) Awight: Document a bit [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346664 [23:41:55] Fundraising-Backlog, FR-Ingenico, FR-WMF-Audit: Ingenico transaction refunded in two partial refunds: watch - https://phabricator.wikimedia.org/T162325#3159266 (awight) [23:54:24] (PS3) Awight: Document a bit [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346664 [23:54:26] (PS1) Awight: Handle commandline param like other fields. [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/346667