[01:06:32] (PS1) Ssmith: WIP don't review: add datepickers/styles [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/148572 [01:06:35] (CR) jenkins-bot: [V: -1] WIP don't review: add datepickers/styles [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/148572 (owner: Ssmith) [01:12:32] (PS2) Ssmith: Modify fraud widget [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/147212 [01:12:55] (PS2) Ssmith: Add popover for SQL query access [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/147648 [01:13:16] (PS2) Ssmith: WIP add datepicker libs and subview [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/147693 [01:38:17] (PS2) Ssmith: Fix requirejs loading + add SQL string to popover [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/148145 [01:38:31] (PS2) Ssmith: Add timepicker subview/make widget header smaller [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/148158 [01:38:42] (PS2) Ssmith: WIP don't review: add datepickers/styles [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/148572 [16:40:21] hello awight! am about to upgrade drupal on staging, but i notice there is no deployment branch for drupal, and the code there is in detached state [16:40:37] awesome! [16:40:43] I donno what you mean by detached? [16:40:56] not on a branch [16:40:58] We haven't needed a deployment branch yet, so just using master [16:41:03] how is... [16:41:14] what is not on a branch, the code on the production server? [16:41:17] * (no branch) [16:41:19] master [16:41:23] where? [16:41:29] lutetium [16:41:33] that's a pretty easy state to get git into and doesn't mean much [16:41:38] ok [16:41:49] is it the same SHA as master? [16:42:05] I think, any git submodule will be detached [16:42:21] ah, i guess that makes sense [16:46:09] hm, local (no branch) is "Merge contrib" 3aadd3a and local master is 9 back. [16:46:33] sorry, you deploy from within the submodule or from the dir above? [16:47:37] from the top-level directory [16:47:38] Also, lutetium has a jenkins instance with no jobs. Does that mean there's nothing feeding into staging civi that I need to stop? [16:47:48] I think that's true [16:47:53] ok, cool [16:48:10] there are cronjobs however... which we cannot stop... but I think they all use the production slave. [16:48:13] (db) [16:48:56] ok. For the real upgrade, do we pester jeff to stop cron jobs? [16:49:25] ejegg: nah, I think we don't run cronjobs on prod [16:49:32] ok [16:49:55] ejegg: do you already have a window for the cookies deployment? [16:50:04] I'm deploying to meta in a few... and could include that. [16:50:30] mwalker wanted to do it in the 2-4pm PST window [16:50:59] oh! how do I verify that the config changes are deployed? [16:51:24] ejegg: right, I see it now--1200-1400 PDT [16:51:32] ejegg: good question. [16:51:36] lemme see... [16:52:05] ejegg: https://noc.wikimedia.org/conf/ [16:52:14] awesome, thanks! [16:53:10] ejegg: it gets messy wrt "wg" vs "wmg", but u must have noticed that already [16:56:22] ok, should be all set. The old ones only existed as "wg", and the new array is in there as a "wg" right next to them [17:13:54] jessicarobell: the-wub: can you please test automatic state changes on the TY letter? [17:13:58] They should be fixed now... [17:16:45] awight: file permissions error doing git pull in /srv/org.wikimedia.civicrm/ unable to unlink sites/all/modules/wmf_common/tests/phpunit/WmfTransactionTest.php [17:17:11] oh erp... one moment, please [17:17:57] ejegg: does that help? [17:18:45] yep! [17:20:29] drush updatedb reports 'no database updates required' - that'll make the real deploy less stressful [17:20:51] (PS1) Awight: Merge remote-tracking branch 'origin/master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/148706 [17:20:58] (CR) Awight: [C: 2] Merge remote-tracking branch 'origin/master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/148706 (owner: Awight) [17:21:51] jessicarobell: sorry, did u see my note about the automatic state changes? Please confirm fix if you have time. [17:23:30] ccogdill: the silverpop thing might be fixed. I'm running an automatic export, the results should be available around 11:45 [17:23:40] Hi awight! No I didn't.. did you email me? thanks [17:23:55] jessicarobell: ah no just IRC but you fell off net at about the same time [17:24:06] well, let me know yay or nay whenever you have the chance, thx! [17:24:29] awesome awight [17:24:33] yea or nay. stupid English. [17:24:56] thanks! I will keep an eye out [17:25:10] yes, I was having some wifi issues. sorry about that. So you just want me to check if it works or not? (Sorry can't find your note..) [17:25:34] awight^ [17:25:35] jessicarobell: yep the translation should become unpublished when you update a paragraph now. [17:25:44] Ok will check. Thanks! [17:25:51] ok, staging looks happy with 7.29. admin/reports/status complains about a writable sites/default/settings.php and missing .htaccess files in priv/pub/tmp dirs though [17:26:17] ejegg: sssh ;) [17:26:24] <_< [17:26:26] >_> [17:36:25] ccogdill it ooks like your ? problem was deployed :) [17:36:36] woo! [17:36:43] I’ll check the data in about an hour then [17:36:46] thanks again [17:42:28] hey jessicarobell - quick question [17:42:38] this card went out with the recurring issues: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1784 [17:42:45] does that mean that sorry... that's an old one [17:42:59] can I delete that one since this one went out? https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1825 [17:58:24] Jeff_Green: can you confirm that the recurring contributions alarm limit is set to a low threshold? And, where is that stuff? [18:01:23] awight: it's a bit opaque [18:01:28] lemme see here [18:03:16] Jeff_Green: I'm gonna try to document those, or u can just dump whatever you find to a wiki page, either way. [18:03:37] jessicarobell: indeed... The state change is not working... [18:03:41] thta would be wise, because I'm having to relearn it here . . . [18:04:24] nope, not yet awight. But we'll get there! Thank you! [18:05:26] jessicarobell: I'll prioritize this, cos I think it's really awkward right now, with the whole manual unpublish/publish workaround. [18:07:02] awight: I'm glad you asked, looks like the check got deactivated somehow [18:07:09] also the raid check for silicon [18:07:17] Jeff_Green: double plus ungood! [18:08:13] it's partially working--the stats are collecting to ganglia [18:08:53] Jeff_Green: so... what IS that stuff? Drupal module? C? [18:09:04] hahahaha [18:09:06] you're funny [18:09:13] no. it's a two part thing [18:09:42] part one is a perl script collect_activemq_stats_via_gmetric [18:10:00] * awight commences irc logging [18:10:04] do tell [18:10:45] that script is in puppet, it calls silicon:/usr/local/apache-activemq/bin/activemq-admin [18:11:03] aha k [18:11:04] and parses the output for a whole bunch of predefined queues [18:11:39] then it injects the results to ganglia using /usr/bin/gmetric [18:12:13] finally it writes a statefile to /tmp/activem_queues.state [18:12:40] second part is a nagios plugin [18:12:59] silicon:/usr/lib/nagios/plugins/check_activemq [18:14:04] that reads the statefile from the ganglia collector, parses for whichever queue you've requested, decided whether or not it is alertworthy (from command line args, per usual nagios plugin usage), and sends the report as a passive check to nagios [18:15:01] so the whole icinga part is working but somehow the nagios config is missing [18:15:07] all of this is puppetized fwiw [18:16:58] https://wikitech.wikimedia.org/wiki/Fundraising/tech/Alerts [18:19:09] ccogdill: ok, the export failed. Back to the shop... [18:19:26] ahh too bad [18:19:32] I wonder why! [18:19:43] ccogdill: was that manual dump good enuf for now? [18:19:57] ccogdill: eh, just a bug, nothing exciting [18:20:04] I think so, but I need to check a few bulk email complaints and make sure it was totally updated [18:20:08] let me get back to you in a few [18:20:36] awight I'm going to cov to my house where the internets is less faily, and I'll fix silicon monitoring today [18:21:01] Jeff_Green: ok rad, no rush on that we just wanted to check up [18:23:11] yeah, I'm perturbed by the fail so i'll rush anyway :-P [18:24:25] atgo-meetingz: did you hear anything about the PayPal fail from this morning's test? [18:24:31] yeah [18:24:34] awight you have some ideas? [18:24:37] i made a card for it [18:24:39] K4 just gave me a heads-up but there is no email. [18:24:43] atgo-meetingz: okthx. [18:25:08] atgo-meetingz: yeah, I'll ned the details cos the going theory is not correct (IPN messages not coming in) [18:25:30] ok i see it. #1829 [18:25:36] https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1829 [18:26:12] ccogdill: r u busy? I want to confirm how many PP donations we've gotten today. [18:26:31] only normally busy, hah :) [18:26:34] hehe. [18:26:36] want me to just look in PP or on civi [18:26:37] ? [18:26:48] ccogdill: the card is above ^^ [18:26:52] ah [18:26:56] they say conversion rates are really low [18:27:04] so I want to compare Civi with the PP console [18:27:23] okay let me take another look [18:27:25] ccogdill: I get 3665 donations since 0000 UTC today [18:27:29] (last 18.5 hours) [18:27:35] according to Civi, yes? [18:27:39] yep [18:27:49] mmkay checking PP [18:27:56] what is ecom? [18:28:18] ejegg: sigh. That's a janky analytics script which we have tried to kill on numerous occasions. [18:28:31] look on prod civi, /srv/br/ecom and fr.py [18:28:40] thanks! [18:29:00] There's a lot of history. [18:29:18] We've rewritten it but for many reasons cannot take the final step and replace that thing. [18:30:09] hey awight are you looking at all currencies or just USD? [18:30:11] sounds like a job for the fr analytics coder that's coming on in the fall [18:31:57] ccogdill: all currencies [18:32:23] ejegg: that's one of the factors keeping us from loosing the dogs on that old code. [18:35:15] meganhernandez: do you have more details about the ecom failure for the last few tests? Pasting me a commandline would be a huge help. [18:35:39] awight I think this is the command [18:35:40] ./ecom -s 20140723000000 -e 20140724000000 --campaign C14_enUS -g p --raw [18:35:45] I’m struggling with PP right now [18:35:48] fyi :) [18:35:50] ccogdill: oh rad, thx [18:36:07] ccogdill: I do not envy that task, thanks for dealing with it [18:36:30] they just make it so hard to get a good report! so silly [18:36:31] ejegg: oh, staging civi server is where to find the scripts actually [18:36:36] no kidding! [18:36:55] ok [18:38:41] argh! little-known space-delimited table formatting [18:42:45] mwalker: Are you deploying CentralNotice for some other reason, or is this just for the cookie? [18:43:19] Cos ejegg and I can take care of the deployment if you'd like. [18:43:28] awight, this is for everything in the backlog -- so all the geoip cookie stuff and the SUL hook [18:43:49] mwalker: that hook will be a no-op for a few weeks still, correct? [18:43:55] I think so yes [18:43:58] * awight looks at backlog [18:44:31] ejegg: CentralNotice$ git cherry -v origin/wmf_deploy origin/master [18:45:12] mwalker: ok well you can just be on hand to test the geoip cookie, or you can do the deploy, whichever you prefer. [18:45:31] if you're willing to deploy it; it's all yours [18:45:52] yep ok. I was not involved in the cookie issue though so do not know how to test. [18:45:59] you want to point me to docs? [18:46:28] ah i found a bugzilla id [18:46:42] https://bugzilla.wikimedia.org/show_bug.cgi?id=66478 [18:46:53] ok, let me reviw the deployment steps [18:48:28] mwalker: okay, so to test, it looks like I should give myself an invalid geoip cookie? [18:49:28] seems like it [18:49:36] I was only involved on the periphery [18:49:43] yah ok thx [18:49:49] but ya; give yourself an invalid; see if it resets to something normal [18:49:53] delete it; make sure you get a new one [18:51:42] awight: we generally take all the localisation updates, right? [18:51:49] ejegg: yep [18:53:06] ccogdill the ? fix didn't take [18:53:27] yeah, Adam said it failed to upload anyway [18:53:37] I just need to be able to pull data manually on Thursday [18:53:52] will my normal script work again or do I have to get a special one? [18:54:07] ccogdill: um. I'll give you the special one. [18:54:15] ccogdill: how fresh do u need the data to be?? [18:54:23] besides those and cookie updates, also taking the 'add known devices' and 'show rtl collapsible arrow' [18:54:37] as fresh as possible :) we had some people receive the Jimmy appeals today after donating yesterday [18:54:47] ejegg: yep the first one is just a first-time db install thing for developers [18:54:51] so pull no earlier than 12pm on Thursday, I’d say [18:54:51] ejegg, add known devices should be a no-op on the cluster so no need to test [18:54:54] ah [18:55:07] ok, cool [18:55:10] ccogdill: ok please ping me later today to set you up with that [18:55:15] and rtl collapsible arrow you can test by setting uselang=he and looking at one of the log tabs [18:55:20] okay, thanks [18:55:52] ejegg: if you want, I could screenshare during deploy [18:56:00] yes, that would be great! [18:56:27] you can watch me fumble [18:56:45] heh. nothing ruins typing skills like an audience [18:57:42] awight re your PP question, neither Pats nor myself can currently get the report we need from PP. we’re waiting for Tony to grant us some permissions [18:57:48] just an update [18:57:49] aaargh. [18:57:52] ok thx [18:58:11] ccogdill: I think I did narrow it down to a different type of error, so I'd say it's only medium-priority to pull those #s [18:58:21] hmm, okay [18:58:25] ccogdill: or, we could sample an hour so we can just count visually. [18:58:36] well I’d like to be able to do it anyway [18:58:42] let me see if I can narrow by time [19:00:35] lunch anyone? [19:00:38] it’s really silly, awight, but I can’t do that either. and PP won’t give me a total number [19:00:51] baaa [19:00:56] ok I am not at all surprised [19:01:10] yeah [19:01:13] rumor is that they want to kill the whole dev access thing [19:01:15] anyway [19:01:24] really? [19:01:25] ugh [19:01:46] well, the API and stuff. they just want lots of money, not to support its acquisition [19:02:07] naturally [19:02:20] their reports can’t be trusted as it is! [19:04:31] (PS1) Awight: Merge remote-tracking branch 'origin/master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/148736 [19:04:41] (CR) Awight: [C: 2] Merge remote-tracking branch 'origin/master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/148736 (owner: Awight) [19:06:03] nobody wants to nom? [19:06:48] atgo: stuck deploying [19:06:54] atgo: I brought nom though [19:06:57] mmk [19:07:04] pizzzacat? ccogdill? [19:07:10] (Merged) jenkins-bot: Merge remote-tracking branch 'origin/master' into wmf_deploy [extensions/CentralNotice] (wmf_deploy) - https://gerrit.wikimedia.org/r/148736 (owner: Awight) [19:07:20] I brought nom too [19:07:37] fiiiine whatever [19:07:38] can’t today atgo [19:07:44] have to buy Tanner’s birthday present :D [19:07:45] haha aww [19:07:53] nice! [19:07:54] good luck [19:07:55] poor atgo-lunch [19:07:58] is it a puppy ccogdill ? [19:08:10] I wish! [19:08:20] I should force a cat on him [19:30:48] pizzzacat: u know much about the cookie debugger in FF? [19:31:09] I'm a chrome person :-/ [19:31:11] I think I have doubles. [19:31:28] poh nvm [19:31:32] wikimedia vs wikipedia [19:31:33] what's happening? [19:31:39] mm [19:31:50] we're looking at a CN deployment to see if the bugfixes are good [19:37:04] still getting old bannerController.js [19:37:09] aha ok [19:37:15] that's caching then [19:37:26] you can probably force with ?debug=true [19:37:46] oh, that was with debug=true, but not logged in [19:38:39] hrmph. Yeah beyond that I think we just have to wait 15 min. [19:38:55] does it have an expiration header? [19:39:31] ejegg: oic what you're saying. Yes, that accounts for the patch not working ;) [19:39:43] Sat, 16 Aug! [19:39:51] It could be a little while [19:40:30] wait, is all the js cached for a month? That can't be right [19:40:56] no I'm looking for the other header... [19:41:33] um... debug=true does not seem to be working. I must be having a stroke. [19:41:53] oh. some RL stuff is still compressed, but the majority is not. [19:41:55] weird. [19:41:56] not working? [19:41:59] OK [19:42:03] RL? [19:42:58] ResourceLoader [19:44:02] ah. Derp, that was the css cached till aug 16th [19:45:03] well... this is why mwalker reserved a 2-hour window. [19:45:06] bannerController.js still says last-modified Jul 10th, and via 1.1 varnish. [19:45:29] so how long does varnish cache js? [19:45:44] 5 minutes [19:45:57] thanks mwalker! [19:46:04] I don't know how last-modified is calculated but I don't think it's related to the actual file [19:46:28] it is exactly what it says it is [19:46:35] try touching the JS file and resyncing [19:46:47] alternatively -- use debug=true and see if you get the newer file [19:47:03] ohhh, bannerController is loaded off of bits.wikimedia.org. Did our update get there? [19:47:27] bits is controlled by resourceloader -- which is a target for sync-dir -- but it sometimes gets confused [19:47:37] which is why touching and resyncing sometimes helps [19:47:45] ooh, complex [19:47:55] ejegg: shite, I just logged into an app server and the js is not updated there... [19:48:02] bah [19:48:03] looking at what happened... argh. [19:48:05] stupid fail. [19:48:18] I updated the submodule pointer but not the submodule code. [19:48:25] hate... git... sometimes [19:48:29] ok [19:48:36] awight, that happens at least twice a week [19:48:38] you're not alone [19:48:47] yesterday it was the wikidata people [19:49:07] so missing command was git submodule update? [19:49:41] ejegg: yep. just pushed the changes. [19:49:53] ok *now* to sit and watch the cache forever. [19:50:32] awight, good news! [19:50:35] got the PP data [19:50:44] awesome. it matches? [19:50:45] 3642 is the total for today [19:50:48] that seems right, yeah? [19:50:52] right on. yep! [19:50:56] thanks again for checking that [19:51:01] mhmm [19:51:22] getting the new js, at least logged in and with debug [19:53:06] and the logged-in guy gets no 'exception' comment in the BannerRandom response from meta [19:53:22] rad! [19:53:30] how is hide? [19:53:50] ejegg: you know how to force a banner? [19:54:14] https://en.wikipedia.org/wiki/Main_Page?banner=B13_0910_notest_dr_yyYY&reset=1&force=1 [19:54:21] "reset" will set the cookies back to 0 [19:54:23] nice, thanks! [19:54:28] "force" overrides any hide checks [19:55:00] "banner" just defeats the random selection [19:57:13] we run on donations averaging about %AVERAGE% [19:57:24] guessing that particular banner is not in a current campaign [19:57:25] hah oh dear [19:57:31] yeah [19:58:39] ejegg: well, the worst-case scenario was that we would display a banner every request. that at least is not happening. [19:59:03] whew! [20:01:39] ejegg: hmm, the cookie is set correctly, but the sister sites just set 'hide' [20:01:49] right, that's expected [20:02:03] ok good [20:02:11] once everybody has js that understands the new format, we can change the server side [20:02:57] nice work! [20:02:59] Oh, we can also add '?reason=donate' to those img src links in the ty page now. [20:03:16] please make a note ;) [20:03:58] hmm, if i use the banner = xyz and debug=true, I get a TypeError: $otherLabel.html(...) is undefined. [20:04:02] trying to track that down [20:07:27] hmm, it's in an eval'ed bit trying to localize currency amounts in the banner [20:07:51] not seeing that code in the repo, guessing it's part of that banner in particular? [20:08:13] ejegg: that looks like [20:08:16] yep you got it [20:08:20] and it's an old banner, so I'm not worried [20:08:26] ok, cool [20:08:32] the-wub (pcoombe) is the person to ask about that [20:08:40] gotcha [20:11:19] so now that this is looking pretty alright, who do I need to consult on scheduling the drupal upgrade on barium? Just mail fr-online? [20:14:56] ejegg: yes good idea. Tell them we'll be down for at most 2 hours, I think. [20:15:06] ejegg: actually, fr-all [20:15:13] ah, cool [20:15:15] cos that includes everyone who uses Civi [20:22:21] ejegg: if you have some down time, there is a terrible review awaiting: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1819 [20:34:16] awight i have a checkin with GR in about 30... nothing i think you'll care about [20:34:40] but if there's anything you want me to bring up/if you want to join, lmk [20:34:44] atgo: ah thx, well feel free to wield the shepherd's crook if u need [20:34:49] ha sounds good! [20:42:05] Jeff_Green: going to send out an email proposing the drupal update sometime on Friday. Was thinking between 10 am PST and 2 pm PST would mean you and awight are both available. Sound good? [20:43:02] will say we want a 2 hr window, and ask whether they prefer earlier (10-12) or later (12-2) [20:43:47] (PS3) Ssmith: WIP don't review: add datepickers/styles [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/148572 [20:43:57] (CR) jenkins-bot: [V: -1] WIP don't review: add datepickers/styles [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/148572 (owner: Ssmith) [20:44:32] Jeff_Green: the IPN switch was just yesterday, right? [20:44:53] I'm trying to track down an IPN issue which started on the 19th... if you know of anything that might have changed then. [20:46:50] awight: yes [20:47:05] ejegg: yep that's ok by me [20:47:16] hrrrmph. [20:47:59] awight: the 19th is roughly when we switched the postback hostname? [20:48:36] i rolled that out everywhere before we switched the listener URL to payments-listener.wm.o [20:48:49] is it? I realize I never made the PP console change, cos I was waiting to confirm we had the "future"-compatible URL [20:49:05] postback... that really cannot be the issue [20:49:07] but good to know [20:49:26] awight: it *looks* like redirects work [20:49:31] what's the issue? [20:49:49] Jeff_Green: eh, the contribution_tracking association has been broken since the 19th [20:50:09] I'm digging around to figure out how the crap that was being done in the first place. [20:50:16] yeah [20:50:33] is that supposed to be done by the qc? or directly by the legacy listener? [20:50:38] donno yet. [20:50:42] ok [20:50:46] the qc runs on barium right? [20:50:54] I think there is a magic number we use and the QC reassociates using that [20:50:57] yeah [20:51:26] ok--so one thing I was wondering...what config does the qc use? [20:51:37] it's all in the db :-/ [20:51:45] ok [20:51:48] or u want to know about specific vars? [20:52:06] just thinking about what my changes were and what they could have broken [20:52:08] hehe this is surreal btw, pizzzacat is being interviewed for ergonomic distress [20:52:19] oh noes [20:52:47] somehow our two cross-interrogations are causing synthesesia [20:53:10] every time I mention config your elbow twitches? [20:53:11] the QC is causing me severe neck pain... [20:53:14] hehehe [20:54:26] well why don't we start with the qc [20:54:50] barium blocks a ton of attempted outbound traffic [20:54:57] i'm not sure what all it is [20:55:37] but I'm suspicious some of it at least is because we're trying to connect to www.paypal.org and we haven't whitelisted all of akamai [20:55:48] I'll keep spelunking to at least figure out what *should* happen [20:56:25] it's a jenkins job? [21:00:17] Jeff_Green: rip contribution_tracking checks out of the PP listener [21:00:23] I did it. [21:00:26] WTF, self [21:00:47] wooo! [21:01:02] I guess it was the fresh deployment which knocked my crappy code into the limelight. [21:01:13] oic [21:01:19] where does the QC run, for future reference? [21:01:26] i don't see anything obvious in barium's jenkins [21:01:34]