[16:54:36] * awight waves ruefully [17:01:25] Jeff_Green: looks like you are preparing to move the banner impressions stuff, today? [17:01:39] i really hope so [17:01:46] i don't really entirely understand it though [17:01:46] OK, I can help. [17:01:49] cool [17:01:52] Yeah it's not a good situation. [17:01:55] ha [17:02:01] so here's what I understand so far [17:02:03] We were hoping to rewrite before anything else [17:02:10] yeah [17:02:18] well maybe we can simplify it a little anyway [17:02:20] so [17:02:23] prolly not [17:02:37] well [17:02:51] so it talks to drupal+civicrm+pgehres db's [17:03:08] distinct db settings for all three, which is handy for testing [17:03:08] grrr [17:03:22] hopefully it's just read on drupal/civi, not sure yet [17:03:23] * awight raises hackles [17:03:28] ha [17:03:39] lucid came with django 1.1.1 [17:03:52] pgehres set up 1.4.1 in a virtualenv under his own user [17:03:55] eff. I thought the django stuff was rolled in [17:03:57] argh ok [17:04:06] and precise (which is barium) is django 1.3.1 [17:04:27] can we just copy over pgehres 's virtualenv directory? [17:04:27] so I'm hoping we can use the stock package and forgo the virtualenv (?) [17:04:34] We can try... [17:04:51] I would rather not do this as a hack copy over [17:04:56] k [17:05:13] as crappy as it may be let's get it in gerrit and normal code deploy for frack [17:05:20] ~pgehres/analytics2, right? [17:05:23] yeah [17:05:42] he's got part of it tied to his own github repo [17:06:14] ok [17:06:14] distribute-0.6.10-py2.6.egg easy-install.pth Pympler-0.2.1-py2.6.egg [17:06:17] Django-1.4.1-py2.6.egg pip-1.2.1-py2.6.egg setuptools.pth [17:06:39] I'm gonna email fr-online to warn that there might be impressions disturbance [17:08:37] ok [17:09:04] But yeah I see your plan is to point at dev_ dbs until we're mostly debugged. Thank you! [17:09:15] hah yeah [17:09:30] i was going to throw it read-only at civi/drupal and write on dev_analytics [17:09:58] perfect [17:10:16] we have to hack it on barium though since that's where the nfs is mounted [17:10:25] which is fine with me [17:10:38] Isn't that its final resting place? [17:10:43] it is yeah [17:11:07] oic, we're bypassing staging. yeah fine w me [17:11:11] so we can hack it in place, and then go back and puppetize and get it in fundraising_code_update [17:11:16] well [17:11:35] i'm not sure how to stage unless we copy over a bunch of log files [17:11:56] i guess we could do that, but the risk of killing things is small until we talk to the wrong db :-) [17:12:46] yeah I'm onboard [17:13:02] let pull this bandaid off quickly :p [17:13:36] I have a meeting 11-12 but otherwise free [17:13:48] alright [17:14:13] q: does it go in an existing gerrit project or a new one? [17:14:20] I think we have one... /me digs [17:14:51] distribute-0.6.10-py2.6.egg easy-install.pth Pympler-0.2.1-py2.6.egg [17:14:54] Django-1.4.1-py2.6.egg pip-1.2.1-py2.6.egg setuptools.pth [17:14:56] urgh [17:14:57] wikimedia/fundraising/tools/DjangoBannerStats [17:15:07] oh fancy [17:15:22] I'll check that we're in sync with pg's github [17:16:18] Jeff_Green: yep, we're ahead of github [17:16:27] great [17:16:33] lemme hack up a config file [17:16:50] that repo is the "analytics2/source" directory [17:16:58] everything else AFAICT is installed by virtualenv. [17:17:18] There are hard assumptions built in that we're running under virtualenv, but I think I can correct [17:17:28] ok [17:18:00] btw, frack login takes ~20 sec today [17:18:03] so we can install it to /srv/DjangoBannerStats on barium [17:18:07] what? not for me [17:18:08] sure [17:18:23] takes me about 5 s to get onto barium [17:18:32] ok 2nd login was 5 sec [17:18:38] odd, i wonder why [17:18:48] dns cache maybe? [17:19:19] Jeff_Green: uh oh, is this a cronjob? [17:19:41] it's currently in jenkins, but I'd like to move it to cron if possible [17:19:52] oh derp. I'm looking at new prod jenkins [17:19:59] why move to cron though? [17:20:08] There's no visibility for the rest of tech, yet [17:20:30] what visibility do you need? [17:20:43] do you actually log into jenkins to tell if its working? [17:20:51] like, is it running, when were failures, configure email, disable/enable... [17:21:05] Yeah totally. What else can I do? [17:21:21] log into indium and read the log? [17:21:42] cron logs every job it fires off [17:21:48] * awight steps back and notices Jeff_Green is a bit older :p [17:21:51] disabling is an issue though [17:21:52] hahahah [17:22:19] I don't actually see the logs on indium... [17:22:23] anyway we can debate that some other time [17:22:30] indium:/var/log/remote/ [17:22:34] yah... [17:22:41] what don't you see? [17:22:49] jenkins is soaking up all your useful debug information [17:22:55] effingjangobannerstats.log [17:23:07] and putting it in a crappy bloated disk schema that I have to prune with handwritten tools :-) [17:23:14] oh [17:23:16] well [17:23:23] Jenkins' ondisk thing makes me furious every time [17:23:33] I especially appreciate that it kills "ls" [17:23:45] i appreciate that it kills itself [17:24:12] without my fine/crappy cron script you would be waiting several minutes to log into jenkins every time [17:24:14] Yeah and the web ui only has the last few logs, cos otherwise Jenkins will hang [17:24:17] yep [17:24:19] right [17:24:23] ok [17:24:27] YAY JAVA [17:24:37] I'm creating barium:/etc/fundraising/django_banner_stats.py [17:24:41] * awight casts piercing laser glances at oracle tower [17:24:52] ok [17:25:23] which is what pgehres has analytics2/source/fundraiser/local_settings.py [17:25:30] grrrrr [17:25:53] thanks for doing that sleuthwork [17:25:55] he just sources is in settings.py [17:26:12] you don't know how happy I will be when this is off of aluminium [17:26:18] that's 3 years in the waiting! [17:26:30] yeah I see it's the last survivor to be rescued [17:26:44] OK the job is not so bad. The virtualenv assumption is limited to one commandline [17:26:45] then we can torpedo the rat infested ship [17:26:51] orly! [17:27:00] hopefully django 1.3.x is passable [17:27:01] source /home/pgehres/analytics2/bin/activate [17:27:01] python /home/pgehres/analytics2/source/manage.py LoadLPImpressions --verbose --recent [17:27:04] python /home/pgehres/analytics2/source/manage.py LoadBannerImpressions2Aggregate --verbose --top --recent [17:27:10] so we just skip the first line [17:27:53] hahaha, he hardcoded UDP_LOG_PATH [17:27:56] oof, moving local_settings.py is rough [17:27:59] UDP_LOG_PATH = "/archive/udplogs/2014/" [17:28:07] that is the best thing in the whole town [17:28:10] why rough? [17:28:20] cos python is not good at absolute path imports [17:28:43] crap and there is some dependency injection happening around settings.py [17:28:54] os.environ.setdefault("DJANGO_SETTINGS_MODULE", "fundraiser.settings") [17:29:14] computing implications... :D [17:29:20] i guess worst case we could symlink it? [17:29:24] but that is stupid [17:29:32] argh. yeah lemme try to avoid that [17:29:50] fundraiser/settings.py itself should be in /etc, IMO [17:30:46] OK easy. we call like PYTHONPATH=/etc/fundraising manage.py ... [17:31:45] (PS1) Awight: Moving the settings to /etc [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/152936 [17:32:00] Jeff_Green: ^^ for you :) [17:32:24] then, we move the settings.py into /etc/fundraising/django_settings.py [17:32:40] And IMO we just concatenate contents of local_settings.py to the end of that file, replacing the try/import [17:32:52] I'll prepare that file on barium... [17:34:12] (CR) Jgreen: [C: 1] Moving the settings to /etc [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/152936 (owner: Awight) [17:34:43] oh should I do that re. django_settings.py ? [17:34:48] I'm on it [17:34:53] k [17:35:20] Jeff_Green: ok, barium:~awight/django_settings.py should install to /etc/fundraising [17:36:47] Jeff_Green: I'll create the job on barium jenk [17:36:50] awight: ok check it out. i redid the db stuff for testing [17:36:51] Unless u really want cron [17:37:19] Looks great, IMO don't use your login but whatever [17:37:24] if it's jenkins I wash my hand [17:37:33] oh that's just temporary, i created it just for this testing [17:37:34] hehehe [17:37:45] I normally don't bother with a mysql user [17:37:51] nice. [17:37:57] one less password to lose :-) [17:38:08] So, I don't actually care which cron we use, but how can the rest of fr-tech turn the thing off in the middle of yr night? [17:38:22] I think there are still issues there that we need to figure out [17:38:26] yeah [17:38:35] maybe we should create a config file that gets rsyncblasted [17:38:51] plus... :( minimizing changes during the migration is probably the best place to start [17:38:51] that gets sourced by anything that cron runs, with a "skip me" deal or something [17:38:54] OK [17:38:57] Let's do that [17:39:03] jenkins is fine for now [17:39:16] although, I don't think nonops currently have permissions on any config files [17:39:32] we can fix that easily [17:39:40] (: [17:39:46] later for that though, lets finish slaying this thing [17:39:49] k [17:40:35] I removed a couple gratuitous entries from that config file [17:40:45] like the blank ADMINS one that just gets clobbered [17:40:56] and the extra DATABASES one [17:41:00] does it look sane now? [17:41:14] Yah I'm sure [17:41:33] Yep. Did you add DEBUG=True? [17:41:44] i didn't [17:41:45] no [17:41:49] hrm. wth is that... [17:41:53] ha [17:42:07] haha: NOTHING [17:42:09] ok then [17:42:17] you have write access on that file atm [17:42:29] ok I will snipe [17:42:41] ok so, now to get the project in fundraising_code_update [17:42:53] moving ADMINS to head cos it is a dependency for other assignments [17:42:59] k [17:43:08] LOL formal title for our herd of cats [17:43:18] done [17:44:02] OK I just saw the UDPLOG 2014, that is rich [17:44:18] spiffy eh? [17:44:30] Can we name it after me, too? [17:44:39] the project? [17:44:44] Adaamantium2014 [17:44:51] oh sure! [17:44:59] * awight strokes frail ego [17:46:00] Jeff_Green: ok the jenkins job is written and disabled [17:46:08] ok [17:46:21] what was the repo path again, i'm failing to find it in backscroll [17:46:35] wikimedia/fundraising/tools/DjangoBannerStats [17:46:40] yep [17:46:42] https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools/DjangoBannerStats [17:47:05] hmm, it doesn't get pulled into tools [17:47:10] I assumed /srv/DjangoBannerStats for the job... I though tyou said that in backscroll and did not see [17:47:14] Oh yeah it's not a submodule [17:47:17] ok [17:47:19] let's... keep it like that [17:47:30] yep [17:47:38] it'll install to /srv/DjangoBannerStats [17:48:01] (CR) Awight: [C: 2] Moving the settings to /etc [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/152936 (owner: Awight) [17:48:07] (CR) Awight: [V: 2] Moving the settings to /etc [wikimedia/fundraising/tools/DjangoBannerStats] - https://gerrit.wikimedia.org/r/152936 (owner: Awight) [17:48:22] right, I keep forgetting that we deny crazies like u +2 [17:48:52] +2 on everything! [17:49:05] aaah the merging, make it stop! [17:50:21] awight just a reminder - MG meeting about first name/last name in 10 [17:50:47] awight: code is deployed [17:51:04] ok i'll run once [17:51:45] Jeff_Green: has logs? [17:51:50] nothing happened. [17:51:58] atgo: yes thanks! [17:52:27] atgo: what you can't see over the internet is that I'm super alert and poised for next meeting :p [17:52:35] haha :P [17:52:49] Mari took notes for me [17:52:51] awight: yeah there are logs [17:53:05] Hrm. OK I might have to table until noon [17:53:12] so close ! [17:53:13] k [17:53:33] there's no db yet [17:53:51] i mean, it's created but empty [17:54:18] I see by dysfunctional ls that 2014 haz files... [17:54:29] voom! [17:54:47] ok, we'll figure it out after the mtg [18:00:17] hi CaitlinVirtue ! [18:00:35] @atgo helloooo! [18:47:34] awight|meet-hed - anything you need from me? [18:47:43] i'm going to make a card for this new piece [18:52:07] awight|meet-hed https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1865 [19:19:34] Jeff_Green: debugging the job now... [19:19:49] k [19:24:28] Hi all.. quick question: what is the proper user-facing name of the comment field to be added in card 746? https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/746 [19:24:29] ooh that's creepy. When I run from the commandline, I get exceptions that didn't appear in the Jenkins logs. [19:24:42] orly [19:24:58] AndyRussG: probably matching the normal mediawiki field... /me looks [19:25:10] In normal MW it's "summary" [19:25:22] yep, that one! [19:25:29] Though in the DB, at least in the recentchanges table, it's "comment" [19:25:34] Although... maybe "change summary" is better [19:25:39] hrm [19:25:54] well I like making it clear that the summary is about the change [19:26:03] Jeff_Green: DatabaseError: (1146, "Table 'dev_analytics.squidlog' doesn't exist") [19:26:12] ah ha [19:26:25] Jeff_Green: fwiw, there is some scary Django stuff, dynamic schema and row insertion... [19:26:27] there's a block of sql there, i wonder if I have to do that by hand [19:26:30] omg [19:26:31] awight: thx! [19:26:40] np! [19:26:49] I'm excited that we'll have this! [19:27:25] Jeff_Green: I'm very concerned about the error not displaying in Jenkins though... [19:27:52] awight: hahahahahHAHHAAHHAhahahahah. perhaps you need to put in more money. [19:28:02] Jeff_Green: 'mail_admins': { [19:28:02] 'level': 'ERROR', [19:28:02] 'filters': [], [19:28:02] 'class': 'django.utils.log.AdminEmailHandler' [19:28:02] } [19:28:06] mebbe... [19:28:15] noooo [19:31:50] ok try again [19:31:55] willdo [19:32:20] Jeff_Green: same nothing in Jenkins. trying cmdline... [19:32:38] perfectly healthy. [19:32:49] sooo. why not from jenkins... [19:33:00] oh [19:33:17] did you copy the jenkins crap from aluminium? [19:33:24] mostly [19:33:33] he had a three-line shell barf for the command [19:33:45] yeah I'm barfing a bit, same number of lines... [19:33:49] different tho [19:33:59] but I'm doing the same thing from the cmdline [19:34:10] It takes 70 seconds. And actually does things. [19:34:18] i wonder if the jenkins user can read the logs [19:34:47] Jeff_Green: no it cannot [19:34:54] SO WHY NO ERRORS [19:34:57] araaaaargh [19:35:05] because...quality code? [19:35:21] There are... some nice things about this code. [19:35:32] did he syslog? [19:35:35] no [19:35:43] maybe django can do that [19:35:47] but I want to not care [19:36:15] maybe it distinguishes between active user and cron? [19:36:57] I think you were right about the jenkins perms [19:37:34] Jeff_Green: btw did u copy my al:~ anywhere? [19:38:06] no, but it's still there [19:38:16] ok I will pack my boxes [19:39:35] ok jenkins should have access now [19:39:43] we might have to restart the daemon for it to know that though [19:40:07] still no nothing. no logs. Yeah let's restart [19:40:20] You want the honors? [19:40:31] yeah doing [19:40:34] done [19:42:50] Jeff_Green: health! [19:43:11] holy mackerels! [19:43:40] Jeff_Green: so, I guess we should stop the old job, then u copy the existing banner_impressions db over [19:43:54] or I guess it's the same db server [19:43:58] well [19:44:06] There... might be some assumptions about filenames :( [19:44:20] making it write to prod is just a matter of changing db settings [19:44:22] looks like the same path! [19:44:25] ok great [19:44:35] yeah path is the same [19:44:43] so that dev_ db was a clone of prod? rad. I think it will work [19:44:50] no it was blank [19:44:54] oh. [19:44:58] in that case, very odd. [19:45:15] It must be limiting reruns to a certain age. [19:45:21] and I ran DjangoBannerStats/sql/create.sql [19:45:23] We should test that on dev, if you don't mind [19:45:33] test which? [19:45:36] just make dev a clone of prod db [19:45:42] oh, ok I can do that [19:45:49] I want to test whether we're going to get duplicate impressions [19:46:05] yep. lemme make sure we have enough room on lutetium [19:46:15] oh. right this is a mammoth [19:46:47] it's better than the one it replaced, just 150GB [19:47:01] the previous incantation was nearly 500GB iirc :-) [19:47:02] well. You know, I'm fine with duplicates :) [19:47:19] let's test it right [19:47:25] baaaa [19:47:28] since we've changed django version too [19:47:29] professionals. [19:47:32] hahahah [19:47:37] FIRE TESTING WITH FIRE [19:47:52] nice [19:51:16] Jeff_Green: Finished moved out of that burning house, fwiw [19:51:27] wait what? [19:51:41] I cleared out things I want from my homedir [19:51:45] oh fab [19:52:02] donno what the retirement home for servers looks like... [19:52:17] i think we're going to move aluminium into frack for now [19:52:21] and wipe it [19:52:23] Jeff_Green: sweet. syslog is not mentioned on https://docs.djangoproject.com/en/1.3/topics/logging/ [19:53:25] http://stackoverflow.com/questions/6205254/how-to-setup-sysloghandler-with-django-1-3-logging-dictionary-configuration [19:53:43] wowza. [19:53:47] Quite the easter egg. [19:53:50] ha [19:54:06] I've come to believe that RTFM is a lie [19:54:28] RTF SO then src [19:54:33] i can never find what I want to know by looking where it is supposed to be [19:54:37] ha yes [19:54:40] then laugh at the poor manual [19:54:49] They tried! There is a webpage! [19:55:23] hehehe 5 releases later, syslog is still not mentioned. [19:55:29] i'm lookiing at puppet config and trying to remember wtf barium:/etc/fundraising/fundstatgent.cfg is for [19:55:43] oh that's for those json files that land on samarium [19:55:56] Memme it could use a comment saying as much, though [19:56:14] ohwow, raw mysql cfg file [19:57:13] it is? [19:57:22] yah, creepy [19:57:27] i seem to have botched the filename [19:57:34] file {'/etc/fundraising/fundstatgent.cfg': [19:57:38] the extra t [19:57:45] it's a real gent [19:57:54] lolth [19:58:28] so what will I break if I fix the filename? [19:59:03] Jeff_Green: it gets the filename from the jenkins job config [19:59:13] /job/Fundraiser Statistics Report Generation/configure [19:59:16] so somebody just worked around my typo? [19:59:22] hehe that would be mwalker|away [19:59:29] :( very |away [20:02:16] ok. i'm going to fix it [20:02:30] ok I'll pause the job [20:02:52] done [20:03:25] k [20:03:41] new filename is there, can you fix the job? [20:04:06] done [20:04:29] you can fire it up at will [20:05:46] i'm adding a comment to puppet at least [20:11:49] (PS3) Ejegg: Canada backend tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/152182 [20:24:13] (PS1) Awight: (FR #1865) Split full name out for Coinbase import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/152963 [20:26:41] ejegg: omg sorry about the extra work [20:27:28] hey, no problem! [20:27:41] I... did not realize. [20:27:46] But nice workaround! [20:27:57] i think those callbacks could be pretty handy [20:28:08] (CR) Awight: [C: 2] Canada backend tests (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/152182 (owner: Ejegg) [20:28:14] (CR) Awight: [C: 2] Allow per-node callbacks in verifyFormOutput [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/152825 (owner: Ejegg) [20:28:23] (Merged) jenkins-bot: Allow per-node callbacks in verifyFormOutput [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/152825 (owner: Ejegg) [20:28:31] (Merged) jenkins-bot: Canada backend tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/152182 (owner: Ejegg) [20:28:47] ejegg: Yeah, I was facing a similar problem with the data provider thing... those are run before the object is instantiated, somehow. [20:29:32] ah, right. [20:29:48] that explains why I had to add those constructor arguments for them [20:30:00] ty for the code review! [20:30:04] of curse [20:30:09] anything else? [20:30:17] hey, are you running debian testing or unstable? [20:30:32] 7.4 [20:30:36]