[00:25:28] <grrrit-wm>	 (PS2) Nuria: Add UAParserUDF from kraken [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata)
[00:36:08] <grrrit-wm>	 (PS3) Nuria: Add UAParserUDF from kraken [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata)
[00:36:23] <grrrit-wm>	 (CR) Nuria: Add UAParserUDF from kraken (16 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata)
[00:41:26] <Ironholds>	 yay!
[03:20:03] <Krinkle>	 Hm..  what's with all the php=zend and stray - and - - and -- 's on this page?
[03:20:14] <Krinkle>	  https://stats.wikimedia.org/wikimedia/squids/SquidReportClients.htm
[07:31:52] <wikibugs>	 Analytics / General/Unknown: Requests for 'undefined' page increasing on wikipedias - https://bugzilla.wikimedia.org/66352#c27 (Jeff Gage) Have we tried contacting Compility?  Report a problem: http://www.complitly.com/Faq.aspx#na36  Opt-out for a site: http://www.complitly.com/Faq.aspx#na44
[08:58:19] <grrrit-wm1>	 (CR) QChris: [C: -2] Add UAParserUDF from kraken (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata)
[10:13:23] <wikibugs>	 Analytics / Refinery: Duplicates/missing logs from esams bits for 2014-09-28T{18,19,20}:xx:xx - https://bugzilla.wikimedia.org/71435 (christian)
[10:13:23] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful - https://bugzilla.wikimedia.org/70085 (christian)
[10:13:23] <wikibugs>	 Analytics / Refinery: Several raw webrequest partitions now marked successful between 2014-10-13T13:xx:xx and 2014-10-13T22:xx:xx - https://bugzilla.wikimedia.org/72028#c2 (christian) The failed partitions between 2014-10-13T15:xx:xx--2014-10-13T21:xx:xx have all exclusively been esams caches. Hence, f...
[10:14:54] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful - https://bugzilla.wikimedia.org/70085 (christian)
[10:14:54] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions for 2014-10-08T1[89]:xx:xx not marked successful - https://bugzilla.wikimedia.org/71881 (christian)
[10:14:54] <wikibugs>	 Analytics / Refinery: Duplicates/missing logs from esams bits for 2014-09-28T{18,19,20}:xx:xx - https://bugzilla.wikimedia.org/71435 (christian)
[10:15:23] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions for 2014-10-07T1[789]:xx:xx not marked successful - https://bugzilla.wikimedia.org/71882 (christian)
[10:15:38] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful - https://bugzilla.wikimedia.org/70085 (christian)
[10:15:38] <wikibugs>	 Analytics / Refinery: Duplicates/missing logs from esams bits for 2014-09-28T{18,19,20}:xx:xx - https://bugzilla.wikimedia.org/71435 (christian)
[10:44:53] <wikibugs>	 Analytics / Refinery: Make webrequest partition validation handle races between time and sequence numbers - https://bugzilla.wikimedia.org/69615#c11 (christian) In terms of severity and other work, it does not line up too well.  And since we're suffering more severe issues, it has not been fixed yet....
[11:22:38] <wikibugs>	 Analytics / Refinery: Duplicates/missing logs from esams bits for 2014-09-28T{18,19,20}:xx:xx - https://bugzilla.wikimedia.org/71435#c2 (christian) It happened again during for the 5 bits partitions from 2014-10-14T16:xx:xx up to and including 2014-10-14T20:xx:xx. Again only esams bits.  Since I've bee...
[14:29:27] <kevinator_cloud>	 Hi milimetric when is this EFF meeting?  I don't have it in my calendar.
[14:29:41] <milimetric>	 kevinator_cloud: today at 10:00 PST
[14:29:45] <milimetric>	 cloud? :)
[14:30:07] <kevinator_cloud>	 yeah, I left my work laptop at work
[14:30:28] <kevinator_cloud>	 I don't have an IRC app on my personal laptop
[14:30:45] <kevinator_cloud>	 so i'm using IRC Cloud
[14:33:08] <grrrit-wm1>	 (PS4) Nuria: Add UAParserUDF from kraken [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata)
[14:35:16] <grrrit-wm1>	 (PS6) Ottomata: Add IpUtil, Geocode, and GeocodeCountryUDF classes [analytics/refinery/source] (otto-geo) - https://gerrit.wikimedia.org/r/164264
[14:43:37] <wikibugs>	 Analytics / Refinery: No new Pagecounts-all-sites files since 2014-10-10 17:00 - https://bugzilla.wikimedia.org/71994#c4 (Andrew Otto) Oh, hm, we only re-submitted the bundles.  Was there a coordinator we had to manually submit too?
[14:50:43] <nuria_>	 taking my daughter to school, will be right back cc milimetric, mforns_
[16:07:38] <wikibugs>	 Analytics / Refinery: Make webrequest partition validation handle races between time and sequence numbers - https://bugzilla.wikimedia.org/69615#c12 (Toby Negrin) ok -- makes sense. Thanks Christian.
[16:07:48] <grrrit-wm1>	 (PS1) Mforns: Use /etc/wikimetrics config files for alembic upgrade [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/166762 (https://bugzilla.wikimedia.org/64294)
[16:19:39] <grrrit-wm1>	 (CR) Milimetric: [C: 2] "this may not feel clean, but it's at least done in the same way as api.wsgi, which is a similar use case. If we find a better way later, " [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/166762 (https://bugzilla.wikimedia.org/64294) (owner: Mforns)
[16:27:45] <grrrit-wm1>	 (CR) QChris: Add UAParserUDF from kraken (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata)
[16:36:38] <wikibugs>	 Analytics / Refinery: Duplicates/missing logs from esams bits for 2014-09-28T{18,19,20}:xx:xx - https://bugzilla.wikimedia.org/71435#c3 (christian) Created attachment 16773   --> https://bugzilla.wikimedia.org/attachment.cgi?id=16773&action=edit kafka.varnishkafka.kafka_drerr.per_second-2014-10-15.png
[16:41:28] <grrrit-wm1>	 (PS1) Milimetric: Adding some more visibility into query problems [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166770
[16:42:10] <grrrit-wm1>	 (PS2) Milimetric: Adding some more visibility into query problems [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166770
[16:42:49] <grrrit-wm1>	 (CR) Milimetric: [C: 2] "self-merging as the system is totally fubar right now and I'm just trying to debug. Will revert any unnecessary stuff." [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166770 (owner: Milimetric)
[17:32:27] <ottomata>	 milimetric: SoS
[17:32:37] <milimetric>	 hi ottomata
[17:32:44] <milimetric>	 i'm in EFF meeting
[17:32:47] <milimetric>	 #torn...
[17:32:53] <ottomata>	 oof, EFF?
[17:55:07] <wikibugs>	 Analytics / Wikimetrics: Alembic upgrade head should not try to access testing DB when running in production - https://bugzilla.wikimedia.org/64294 (Marcel Ruiz Forns) PATC>RESO/FIX
[18:05:31] <grrrit-wm1>	 (PS1) Milimetric: Stop executing broken queries [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166782
[18:06:01] <grrrit-wm1>	 (CR) Milimetric: [C: 2] "experimenting through self-merges is so wrong..." [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166782 (owner: Milimetric)
[18:17:05] <ottomata>	 qchris_meeting: getting a .deb ready for webstats depoy'
[18:17:06] <ottomata>	 still wanna?
[18:17:48] <qchris>	 Just in time ...
[18:17:57] <qchris>	 The EFF meeting went longer than expected.
[18:18:01] <qchris>	 Yes! Totally wanna.
[18:18:08] <ottomata>	 (what is EFF?)
[18:18:10] <ottomata>	 oH
[18:18:12] <ottomata>	 EFF
[18:18:13] <ottomata>	 meeting
[18:18:14] <nuria_>	 ottomata: every query I do on this 'date' range select 'whatever' from webrequest where year=2014 and month=9 and day=1 and hour=12 limit 100;
[18:18:16] <ottomata>	 that EFF
[18:18:20] <ottomata>	 COOL
[18:18:24] <ottomata>	 man, i woulda joined that :)
[18:18:26] <ottomata>	 forgot about it
[18:18:37] <qchris>	 I'll ping you next time then.
[18:18:41] <ottomata>	 heh, k!
[18:18:51] <ottomata>	 nuria
[18:18:53] <ottomata>	 we only keep 31 days
[18:18:57] <ottomata>	 do
[18:18:57] <nuria_>	 ottomata: fails with an exception, i looked at logs but exception is the same than the command line one so not much info
[18:19:05] <ottomata>	 show partitions webrequest;
[18:19:24] <nuria_>	 ah so "java.lang.reflect.InvocationTargetException" -> no data
[18:19:29] <nuria_>	 otoomata: ok
[18:19:29] <ottomata>	 i guess so?
[18:19:31] <ottomata>	 ha
[18:19:47] <qchris>	 Yes. That's the exception you get, when no partitions matched.
[18:20:24] <qchris>	 ottomata: About the debs ...if we roll them out right now, I guess we'll ruin the data for this hour.
[18:20:37] <qchris>	 Could we roll out around 19:00
[18:20:43] <qchris>	 After the file has been written?
[18:20:44] <ottomata>	 sure.
[18:20:50] <qchris>	 Aaaaaaaaawesome!
[18:21:02] <ottomata>	 ping me when time approaches?
[18:21:07] <ottomata>	 in case i get busy?
[18:21:07] <ottomata>	 :)
[18:21:10] <qchris>	 Like 5 minutes before?
[18:21:22] <ottomata>	 10 mins before, i will need to stop puppet,  add to apt, etc.
[18:21:27] <qchris>	 k.
[18:21:30] <ottomata>	 danke
[18:21:30] <qchris>	 Alarm clock set.
[18:21:33] <ottomata>	 :)
[18:21:35] <qchris>	 tick. tick. tick....
[18:21:42] <nuria_>	 ottomata, qchris: good, then the ua udf in hive is working well. Will submit another patch.
[18:21:44] <ottomata>	 qchris-alarm-clock.activate()
[18:22:01] <qchris>	 nuria_: \o/
[18:50:00] <qchris>	 ottomata: qchris-alarm-clock.callback()
[18:50:02] <qchris>	 ALARM!
[18:50:02] <qchris>	 ALARM!
[18:50:03] <qchris>	 ALARM!
[18:50:32] <qchris>	 Don't dare to punch me into "snooze"-mode!
[18:50:33] <ottomata>	 heheh
[18:52:13] <Ironholds>	 we need qchrisfacts.org
[18:52:15] <ottomata>	 ok, qchris, so, we need to deploy to oxygen and gadolinium, correct/
[18:52:27] <Ironholds>	 Behind qchris's beard: IS ANOTHER BEARD.
[18:52:28] <qchris>	 yes. I think so.
[18:52:42] <Ironholds>	 Icinga alerts do not happen to qchris. qchris happens to Icinga.
[18:53:01] <ottomata>	 k
[18:53:06] <ottomata>	 ok, disabled puppet on those machines
[18:53:08] <ottomata>	 adding 0.4 to apt
[19:00:17] <ottomata>	 ok, qchris, i see the 1900 file
[19:00:23] <qchris>	 Cool.
[19:00:23] <ottomata>	 ok to stop collector?
[19:00:38] <ottomata>	 doing it.
[19:00:39] <qchris>	 When the files are no longer increasing
[19:00:42] <qchris>	 Yes.
[19:00:46] <ottomata>	 increasing?
[19:00:48] <ottomata>	 the tmp file was moved.
[19:00:50] <qchris>	 In size.
[19:00:53] <qchris>	 Ok.
[19:01:17] <qchris>	 Do it :-)
[19:01:56] <ottomata>	 collector 0.4 running, restarting udp2log witih 0.4 filter on oxygen now...
[19:04:32] <qchris>	 New filter seems to be running.
[19:04:41] <qchris>	 New collector too.
[19:04:49] <ottomata>	 cool.
[19:05:19] <grrrit-wm1>	 (PS1) Milimetric: Fix query's cursor descriptor [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166797
[19:05:21] <ottomata>	 qchris, should we merge the refinery changes and restart the job?
[19:05:24] <qchris>	 So I guess we'll see in an hour whether it worked.
[19:05:33] <qchris>	 Yes, we should merge them.
[19:05:35] <ottomata>	 just htis?
[19:05:37] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/165725/
[19:05:38] <ottomata>	 right?
[19:05:40] <grrrit-wm1>	 (CR) Milimetric: [C: 2 V: 2] Fix query's cursor descriptor [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166797 (owner: Milimetric)
[19:06:09] <qchris>	 About restarting ... I would do that it some hours to match the switch for the C implementation.
[19:06:24] * qchris checks the change number
[19:06:54] <qchris>	 The one you mentioned above, and also
[19:06:56] <qchris>	 https://gerrit.wikimedia.org/r/#/c/165378/
[19:08:57] <ottomata>	 ah right, ok
[19:09:00] <ottomata>	 so, i will merge now
[19:09:08] <ottomata>	 hm, qchris, will an hdfs deployment autmoatically pick this up?
[19:09:11] <ottomata>	 it will, right/
[19:09:14] <ottomata>	 because of current?
[19:09:15] <qchris>	 Right.
[19:09:28] <qchris>	 So we should wait a bit with the deployment
[19:09:31] <ottomata>	 ok, welp, not good, but easier for me to deploy right now :)
[19:09:31] <ottomata>	 ok
[19:09:36] <ottomata>	 merging now though.
[19:09:38] <grrrit-wm1>	 (PS2) Ottomata: Stop considering requests for 'undefined' and 'Undefined' [analytics/refinery] - https://gerrit.wikimedia.org/r/165378 (https://bugzilla.wikimedia.org/66352) (owner: QChris)
[19:09:39] <qchris>	 Cool.
[19:09:47] <grrrit-wm1>	 (CR) Ottomata: [V: 2] Stop counting 301, 302, 303 HTTP status codes [analytics/refinery] - https://gerrit.wikimedia.org/r/165725 (https://bugzilla.wikimedia.org/71790) (owner: QChris)
[19:09:59] <grrrit-wm1>	 (CR) Ottomata: [V: 2] Stop considering requests for 'undefined' and 'Undefined' [analytics/refinery] - https://gerrit.wikimedia.org/r/165378 (https://bugzilla.wikimedia.org/66352) (owner: QChris)
[19:10:13] <ottomata>	 ok, qchris, i think you can deploy this!
[19:10:14] <ottomata>	 ja?
[19:10:19] <qchris>	 Yup.
[19:10:21] <qchris>	 Thanks!
[19:11:18] <grrrit-wm1>	 (PS5) Nuria: Add UAParserUDF from kraken [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata)
[19:12:25] <wikibugs>	 Analytics / Visualization: Limn ignores timespan: {start: ..., end: ..., step: ...} configuration - https://bugzilla.wikimedia.org/72088 (Dan Andreescu) NEW p:Unprio s:normal a:None if a datasource defines a timespan entry, the timeseries metric should respect it and use it to timebox the q...
[19:17:24] <grrrit-wm1>	 (PS1) Milimetric: Fix bad timespan for graph [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166798
[19:17:38] <grrrit-wm1>	 (CR) Milimetric: [C: 2] Fix bad timespan for graph [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166798 (owner: Milimetric)
[19:20:07] <grrrit-wm1>	 (PS2) QChris: Stop counting 301, 302, 303 HTTP status codes [analytics/refinery] - https://gerrit.wikimedia.org/r/165725 (https://bugzilla.wikimedia.org/71790)
[19:21:33] <qchris>	 ottomata: the second of the two commits was hanging, as only the first one got rebased. I now rebased the second one. Is it ok to merge?
[19:22:30] <ottomata>	 yes
[19:22:30] <ottomata>	 sorry
[19:22:32] <ottomata>	 didn't see that
[19:22:53] <grrrit-wm1>	 (CR) QChris: [V: 2] Stop counting 301, 302, 303 HTTP status codes [analytics/refinery] - https://gerrit.wikimedia.org/r/165725 (https://bugzilla.wikimedia.org/71790) (owner: QChris)
[19:23:00] <qchris>	 No worries. Thanks.
[19:28:52] <yurikR>	 ottomata, back next week?
[19:31:40] <ottomata>	 yurikR: ja, will be back in town on sunday eve i think
[19:31:55] <yurikR>	 ottomata, cool, mon @ coworking?
[19:33:24] <wikibugs>	 Analytics / Wikimetrics: Story: User creates cohort with CentralAuth insertions - https://bugzilla.wikimedia.org/66843 (Marcel Ruiz Forns) NEW>ASSI a:Marcel Ruiz Forns
[19:34:16] <ottomata>	 yurikR: let's do tues!
[19:34:27] <yurikR>	 ottomata, awesome!
[19:34:51] <ottomata>	 got some morning meetings, so, can be there around noon or 1
[19:37:23] <yurikR>	 ok
[19:56:56] <qchris>	 ottomata: if tin does not let me in ("Permission denied (publickey)."), is it ok to just deploy from my own checkout? (I did pull the git-fat files)
[19:57:14] <ottomata>	 can you do that?
[19:57:20] <ottomata>	 hm, i suppose, but why does tin not let you in?
[19:57:22] <ottomata>	 tin.eqiad.wmnet?
[19:57:25] <qchris>	 Right.
[19:57:27] <ottomata>	 hm
[19:57:34] <ottomata>	 have you gotten into tin befor?
[19:57:43] <qchris>	 but I am neither in deployment, parsoid-admin, or ocg-render-admins groups.
[19:57:47] <ottomata>	 hm, weird
[19:57:47] <qchris>	 No, I haven't
[19:57:48] <ottomata>	 we should get you in!
[19:57:51] <ottomata>	 i will deploy from tin for now
[19:57:55] <ottomata>	 you can do hdfs deploy
[19:58:07] <qchris>	 Ok. But we should still wait a bit :-)
[19:58:11] <qchris>	 I am just preparing.
[19:58:33] <ottomata>	 ja, well, git deploy is fine
[19:58:37] <ottomata>	 just did that
[19:58:41] <ottomata>	 you can deploy to hdfs when you are ready
[19:59:08] <qchris>	 ok.
[19:59:13] <qchris>	 Thanks.
[20:10:38] <grrrit-wm1>	 (PS1) Jdlrobson: Enable edits-monthly-new-active remove edits-monthly-active-2014 [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166859
[20:13:15] <grrrit-wm1>	 (CR) Milimetric: [C: -1] Enable edits-monthly-new-active remove edits-monthly-active-2014 (2 comments) [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166859 (owner: Jdlrobson)
[20:14:34] <grrrit-wm1>	 (PS2) Jdlrobson: Enable edits-monthly-new-active remove edits-monthly-active-2014 [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166859
[20:14:36] <grrrit-wm1>	 (CR) jenkins-bot: [V: -1] Enable edits-monthly-new-active remove edits-monthly-active-2014 [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166859 (owner: Jdlrobson)
[20:17:14] <grrrit-wm1>	 (PS3) Jdlrobson: Enable edits-monthly-new-active remove edits-monthly-active-2014 [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166859
[20:22:48] <grrrit-wm1>	 (CR) Milimetric: [C: 2] Enable edits-monthly-new-active remove edits-monthly-active-2014 [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/166859 (owner: Jdlrobson)
[20:23:18] <ottomata>	 sigh	qchris, udp2log on oxygen needed a second kick
[20:23:41] <ottomata>	 i think it wasn't running now since i restarted it earlier
[20:23:57] <qchris>	 ?
[20:24:00] <qchris>	 Ouch!
[20:24:11] <ottomata>	 yeah, i restarted it, and everything looked fine
[20:24:24] <ottomata>	 i just double checked it (beacuse i got an alert about the udp2log-lucene instance on oxygen) and it wasn't running
[20:24:24] <qchris>	 Ja. ... I saw the filter process running :-/
[20:24:32] <ottomata>	 i thought i did too
[20:25:19] <qchris>	 but ... lucene's udp2log is separate ... isn't it?
[20:25:27] <qchris>	 Mhmm.
[20:25:27] <ottomata>	 i yes
[20:39:43] <nuria>	 ottomata: is TABLESAMPLE something we can use in our dataset on hive or do tables have to be build ina special way to use it: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Sampling
[20:41:15] <qchris>	 nuria: you can use it on every table. but some table samplings are unfair.
[20:41:30] <qchris>	 https://gerrit.wikimedia.org/r/#/c/162589/2/oozie/webrequest/generate_tsvs/generate_sampled-1000_tsv.hql
[20:41:42] <qchris>	 ^ line 62 is a fair sampling.
[20:42:26] <nuria>	 hive (wmf_raw)> select * from webrequest TABLESAMPLE (100 ROWS) ;
[20:42:26] <nuria>	 FAILED: SemanticException [Error 10041]: No partition predicate found for Alias "webrequest" Table "webrequest"
[20:42:31] <nuria>	 ^ qchris
[20:42:45] <qchris>	 add a WHERE clause for the partitions.
[20:43:24] <qchris>	 like "WHERE webrequest_source='mobile' AND year=2014 AND month=10 AND day=14 AND hour=10"
[20:43:46] <nuria>	 tried that before select * from webrequest where year=2014 and month=10 and day=1 and hour=12 TABLESAMPLE (100 ROWS) ;
[20:43:52] <nuria>	 a wait qchris
[20:44:02] <nuria>	 maybe the table sample goes before teh where
[20:44:04] <nuria>	 *the
[20:44:05] <qchris>	 yes.
[20:44:07] <qchris>	 totally.
[20:46:03] <Ironholds>	 and you need an alias
[20:46:26] <Ironholds>	 SELECT * FROM webrequest TABLESAMPLE(100 ROWS) tab_alias WHERE tab_alias.year = 2014 AND...
[20:46:30] <qchris>	 no.
[20:46:34] <Ironholds>	 at least: when I've used it, it's shouted unless I've used an alias
[20:46:37] <Ironholds>	 really? cool!
[20:46:43] * Ironholds makes his HQL files more readable.
[20:46:46] <qchris>	 alias is only needed for subqueries.
[20:46:53] <qchris>	 https://gerrit.wikimedia.org/r/#/c/162589/2/oozie/webrequest/generate_tsvs/generate_sampled-1000_tsv.hql
[20:46:59] <qchris>	 ^ worked as is ... without an alias.
[20:52:11] <nuria>	 mforns: i saw you grabbed teh centrauth bug
[20:52:18] <mforns>	 yes
[20:52:32] <mforns>	 dan recommended it to me
[20:52:35] <nuria>	 mforns: one thing you need to do is to enable teh centralauth role in vagrant so you have the right db
[20:52:55] <mforns>	 ok
[20:53:04] <nuria>	 mforns: be sure to do that or otherwise you do not have centrauth db to acccess
[20:53:23] <nuria>	 that ticket needs also puppet work to setup a testing db etec
[20:53:28] <nuria>	 *etc
[20:53:43] <nuria>	 mforns: we can talk about that once you have the role set up if you want
[20:53:49] <mforns>	 ok
[20:55:49] <nuria>	 mforns: how to setup roles in vagrant is explained in mediawiki docs
[20:56:33] <nuria>	 mforns: after setting up centralauth you probably want to do vagrant halt/up to make sure there are no issues
[20:59:29] <qchris>	 !log deployed refinery ae72e8952b8c776c5cb298b833129f78e662595a to hdfs
[21:39:14] <ottomata>	 ok, off for the day all
[21:39:26] <ottomata>	 qchris, feel free to SMS me if there are any webstats problems
[21:39:31] <ottomata>	 i'll be close to compy
[21:39:36] <qchris>	 Ok. Thanks.
[21:39:44] <qchris>	 But I guess it won't be needed.
[21:39:46] <qchris>	 Have fun!
[21:39:51] <ottomata>	 latesr!@
[22:38:58] <dr0ptp4kt>	 Ironholds: you thre?
[22:53:55] <wikibugs>	 Analytics / General/Unknown: Inherent dropping of messages during deployment of webstatscollector on 2014-10-15 - https://bugzilla.wikimedia.org/72102 (christian) NEW p:Unprio s:normal a:None In order to redeploy webstatscollector, both the filter and the collector process need to be restar...
[22:55:08] <wikibugs>	 Analytics / General/Unknown: Inherent dropping of messages during deployment of webstatscollector on 2014-10-15 - https://bugzilla.wikimedia.org/72102#c1 (christian) NEW>RESO/WON Note that this only affects the files in    http://dumps.wikimedia.org/other/pagecounts-raw  . The files in   http://d...
[22:55:23] <wikibugs>	 Analytics / General/Unknown: Requests for 'undefined' page increasing on wikipedias - https://bugzilla.wikimedia.org/66352 (christian)
[22:55:23] <wikibugs>	 Analytics / General/Unknown: By counting HTTP redirects, webstatscollector reporting too high numbers - https://bugzilla.wikimedia.org/71790 (christian)
[23:02:40] <Ironholds>	 dr0ptp4kt, yep
[23:02:43] <Ironholds>	 what's up?
[23:03:29] <dr0ptp4kt>	 Ironholds: i was looking to do stuff correctly with hql. are you available for a ghangout?
[23:03:41] <dr0ptp4kt>	 or are you going to bed soon?
[23:04:18] <Ironholds>	 neither at the moment, I'm doing some fairly urgent work for the POODLE breach
[23:04:37] <Ironholds>	 but if you have specific questions I'm happy to answer them via text, albeit with a small time-delay
[23:04:40] <dr0ptp4kt>	 Ironholds: goto POODLE; // maybe later...
[23:05:00] <Ironholds>	 (also, can we stop giving security problems adorable names? It makes it really hard to explain to friends and such. "sorry, can't hang out, gotta work on the poodle")
[23:05:56] <dr0ptp4kt>	 Ironholds: :)
[23:10:10] <Ironholds>	 dr0ptp4kt, okay, email em to me? I don't want to drop the ball ;p
[23:10:52] <dr0ptp4kt>	 Ironholds: i'm going to tinker a bit more, and if i can't figure it out, will do.
[23:10:53] <wikibugs>	 Analytics / General/Unknown: By counting HTTP redirects, webstatscollector reporting too high numbers - https://bugzilla.wikimedia.org/71790#c12 (christian) PATC>RESO/FIX Fix has been deployed on 2014-10-15 ~19:01 and is effective.  The last affected files are    http://dumps.wikimedia.org/other/...
[23:11:37] <wikibugs>	 Analytics / General/Unknown: Requests for 'undefined' page increasing on wikipedias - https://bugzilla.wikimedia.org/66352#c29 (christian) Fix has been deployed on 2014-10-15 ~19:01 and is effective.  The last affected files are    http://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-10/pagecounts...
[23:12:08] <wikibugs>	 Analytics / General/Unknown: Requests for 'undefined' page increasing on wikipedias - https://bugzilla.wikimedia.org/66352 (christian) PATC>RESO/FIX
[23:14:05] <Ironholds>	 cool!
[23:19:52] <wikibugs>	 Analytics / General/Unknown: By counting HTTP redirects, webstatscollector reporting too high numbers - https://bugzilla.wikimedia.org/71790#c13 (Nemo) Are retroactive adjustments of stats.wikimedia.org pageview stats expected?
[23:21:54] <wikibugs>	 Analytics / General/Unknown: Inherent dropping of messages during restart of webstatscollector filter on 2014-10-15 - https://bugzilla.wikimedia.org/72107 (christian) NEW p:Unprio s:normal a:None After the deployment of webstatscollector's filter on 2014-10-15, udp2log-lucene did not restar...
[23:22:41] <wikibugs>	 Analytics / General/Unknown: Inherent dropping of messages during restart of webstatscollector filter on 2014-10-15 - https://bugzilla.wikimedia.org/72107#c1 (christian) NEW>RESO/WON Note that this only affects the files in    http://dumps.wikimedia.org/other/pagecounts-raw  . The files in   http...
[23:23:07] <wikibugs>	 Analytics / General/Unknown: Inherent dropping of messages during restart of webstatscollector filter on 2014-10-15 - https://bugzilla.wikimedia.org/72107 (christian)
[23:23:08] <wikibugs>	 Analytics / General/Unknown: Inherent dropping of messages during deployment of webstatscollector on 2014-10-15 - https://bugzilla.wikimedia.org/72102 (christian)
[23:25:13] <grrrit-wm1>	 (CR) Nuria: "Just noting that I tested this in vagrant a bunch." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/166762 (https://bugzilla.wikimedia.org/64294) (owner: Mforns)
[23:44:00] <Ironholds>	 now here's a quick question for qchris, if he's still around
[23:44:15] <qchris>	 Sleepy and about to hit "send" on an email to you.
[23:44:17] <Ironholds>	 can you think of any good reason why grepping for https in the URL field in the sampled logs would get me nothing?
[23:44:21] <Ironholds>	 aha! Cool :)
[23:44:37] <qchris>	 Well ... sampled logs don't get traffic from ssl terminators
[23:44:45] <Ironholds>	 ...bollocks. Okay, that'd do it. Thanks.
[23:44:52] <qchris>	 There's even a bug to document that IIRC.
[23:45:07] <Ironholds>	 *nods*. Was this the overlapping ranges problem?
[23:45:22] <Ironholds>	 no, nevermind, I'm misremembering. Okay, thanks!
[23:46:41] <qchris>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=60315
[23:46:44] <qchris>	 Ironholds: ^
[23:46:55] <qchris>	 No, it's just the logging setup.
[23:47:14] <qchris>	 The nginxes are pretty separate, so they don't make it into sampled-1000.
[23:47:25] <qchris>	 It's not connected to the overlapping ranges.
[23:47:57] <Ironholds>	 *nods*
[23:48:06] <qchris>	 So what you can do to extract the traffic from the sampled logs nonetheless, is checking the client IP.
[23:48:24] <qchris>	 If the client IP is an ssl terminator, it is an ssl request.
[23:48:48] <qchris>	 (At least for clients that did not mess really, really, really hard with their environment)
[23:49:41] <qchris>	 (That's actually what I did to produce this data for the POODLE thread)
[23:50:15] <Ironholds>	 aha!
[23:51:19] <qchris>	 On stat1002 /home/qchris/tmp/2014-10-15-ssl/filter.sh might help with the regexp.