[07:02:24] morning :) [07:02:38] just suspended the aqs-hourly coordinator [07:06:03] I'd like to do hdfs dfs -chown -R analytics:analytics /wmf/data/wmf/aqs [07:06:14] but better to wait for somebody to confirm this :) [07:06:19] (re-enabling the coord) [07:07:52] Morning elukey [07:13:25] bonjour! [07:17:02] joal: whenever you are caffeinated lemme know if my proposal to chown analytics:analytics is ok for you (also related to what I wrote in the email to the team) [07:17:27] I'm reading it now, with café indeed :) [07:21:52] elukey: the plan looks good to me :) [07:23:51] \o/ [07:24:09] ok so let's chown the aqs files and see? hdfs dfs -chown -R analytics:analytics /wmf/data/wmf/aqs ? [07:24:57] elukey: with a sudo I guess? [07:30:55] yep yep [07:31:08] sudo -u analytics should be enough [07:31:19] right [07:32:14] ok proceeding [07:32:18] suspended aqs-hourly-coord as precaution [07:41:15] (03CR) 10Joal: Remove wikipedia-zero as program is over (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507933 (https://phabricator.wikimedia.org/T213770) (owner: 10Joal) [07:41:46] done! [07:42:25] !log chown -R /wmf/data/wmf/aqs/* to analytics:analytics (was: analytics:hdfs) [07:42:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:42:36] let's check the next jobs :) [07:43:42] elukey: I guess you've resumed the aqs job [07:44:12] yep! [07:44:20] ack! [07:55:25] (03CR) 10Elukey: [C: 03+2] Move aqs hourly to the new 'analytics' user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507743 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [07:55:33] (03CR) 10Elukey: [V: 03+2 C: 03+2] Move aqs hourly to the new 'analytics' user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507743 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [08:06:00] (03PS1) 10Elukey: banner_activity: move oozie coordinators to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508283 (https://phabricator.wikimedia.org/T220971) [08:07:52] joal: as migration strategy, I vote for following the order of directories in /wmf/data/wmf/ :D [08:09:01] works for me elukey :) [08:10:50] banner activity should be straightforward to move [08:53:04] analytics:analytics seems to work fine! [08:54:44] joal: asking permission to move banner_impressions as well [08:54:55] (kill, chown, start) [08:55:04] Yessir! [08:55:24] elukey: you'll be more expert than me on oozie jobs after the move! [08:55:40] yeah sure! -.- :D [09:12:11] 10Analytics, 10Analytics-Kanban, 10Operations, 10Traffic, 10Patch-For-Review: Add prometheus metrics for varnishkafka instances running on caching hosts - https://phabricator.wikimedia.org/T196066 (10fgiunchedi) >>! In T196066#5155909, @Ottomata wrote: > I don't think Magnus would build it into librdkafk... [09:13:53] !log kill banner impression coordinators, chown /wmf/data/wmf/banner_impressions to analyits:analytics and start coordinators again [09:13:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:15:32] joal: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0029404-180510140726946-oozie-oozi-C/ looks a bit weird to me [09:15:53] indeed [09:16:22] elukey: date issue I guess? [09:16:57] I haven't restarted it yet [09:17:11] wow :) [09:17:15] yeah :D [09:18:42] elukey: druid confirms - daily segments instead of monthly starting 2018-06 [09:18:47] Mwarf :( [09:19:24] INVESTIGATION ! [09:19:38] they have all timed out [09:19:56] elukey: yeah, but coord says timeout = -1 [09:20:48] elukey: will restart it with old dates up to 2019 (to clean 2018) [09:20:49] ok? [09:21:16] sure! [09:22:06] do we need to do it since there is not data for 2018? [09:22:34] (asking to understand how to fix/investigate these) [09:25:15] elukey: this job is particular - it uses data stored in druid to reindex it in druid itself [09:25:30] elukey: segment-bundling [09:25:43] elukey: I also think we should update this job to sent an email if it fails [09:28:03] ahhh ok so druid contains daily segments [09:28:09] after 2018-06 [09:28:20] (as opposed to monthly) [09:28:35] !log Launch new banner_activity-druid-monthly-coord between 2018-06-01/2018-12-31 to cover for timedout past actions [09:28:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:28:42] correct elukey [09:29:37] Ahhhhh - I think I have an idea :) [09:30:16] elukey: the monthly job needs a 3 month old SUCCESS file in a daily-folder [09:30:33] BUT - daily folders are deleted after 90 days [09:30:41] --> No success file, eternal wait [09:30:47] ooook [09:30:54] We need to find a way [09:31:03] I didn't get the 3 month old SUCCESS :D [09:31:51] elukey: in the coordinator, there is data-dependency for 89 days [09:32:02] actually 90 days [09:32:04] yep but I didn't get why.. [09:32:20] I thought it needed only the past month of (daily) data [09:32:34] elukey: the comment says it sanitizes data, so I guess that's the thing [09:32:47] ahhhhhh okok [09:32:59] !log manually touching success files to start banner_activity-druid-monthly-coord between 2018-06-01/2018-12-31 [09:33:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:05:40] 10Analytics: Aggregate pageviews to Wikidata entities - https://phabricator.wikimedia.org/T215438 (10Sascha) Friendly ping? Meanwhile, I’ve built an ad-hoc “pipeline” for computing this on a laptop. But I still think that having this metric would be generally useful for all kinds of projects, both for users of... [11:07:57] joal: sorry got distracted by another task.. can I kill the banner impression daily ? [11:08:04] not sure the status of monthly [11:12:39] Hi elukey [11:14:05] monthly is catching baxk on the date I recreated _SUCCESS files - We need to rework it [11:15:21] ah on hdfs I have only "daily" partitions right? [11:15:38] so I can kill the daily coord, chmod, and then restart only that one [11:15:44] leaving the monthly aside? [11:16:03] correct elukey [11:16:21] Creating a task for refactor of monthly [11:19:34] joal: just chowned data and relaunched the daily [11:19:36] seems good [11:19:42] awesome :) [11:22:42] 10Analytics, 10Analytics-Kanban: Fix oozie banner_impression monthly job - https://phabricator.wikimedia.org/T222603 (10JAllemandou) [11:23:12] 10Analytics, 10Analytics-Kanban: Fix oozie banner_impression monthly job - https://phabricator.wikimedia.org/T222603 (10JAllemandou) A manual fix has been applied to 2018 jobs. [11:23:46] 10Analytics, 10Analytics-Kanban: Fix oozie banner_impression monthly job - https://phabricator.wikimedia.org/T222603 (10JAllemandou) [11:26:29] (03PS1) 10Elukey: browser-general-coord: move oozie job to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508307 (https://phabricator.wikimedia.org/T220971) [11:26:38] joal: moving to browser-general if you are ok [11:27:32] Yessir - IRRC this one is a bundle, therefore 2 folders to chwon [11:28:09] ah I thought it was a coordinator [11:28:23] don't see the bundle properties [11:31:31] joal: are you sure it is a bundle? [11:36:30] going to eat something for lunch, will be back in a bit :) [12:15:12] !log kill browser-general-coor, chown analytics:analytics /wmf/data/wmf/browser, restart coordinator with user 'analytics' [12:15:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:18:44] (brb) [12:35:16] Wow my bad elukey - browser general is indeed a coord - It was a bundle in my mind- my apologizes [12:40:09] o/ [12:41:59] joal: np! You can't possibly remember by heart all the coords! (well you almost do to be honest, this proves your not a bot! :) [12:46:25] (03PS1) 10Elukey: Move oozie coords interlanguage/mediacounts to the new user analytics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508314 (https://phabricator.wikimedia.org/T220971) [12:47:23] (03PS2) 10Elukey: Move oozie coords interlanguage/mediacounts to the new user analytics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508314 (https://phabricator.wikimedia.org/T220971) [12:48:31] joal: ok to move interlanguage/mediacounts ? [12:49:01] Yep :) [12:49:18] \o/ [12:49:26] I am checking perms etc.. just to be sure [12:49:33] but they seem really straightforward [12:49:35] yessir [12:49:40] I am creating a patch for each step that I take [12:52:12] !log kill interlanguage-coord, chown analytics:analytics /wmf/data/wmf/interlanguage, restart coordinator with user 'analytics' [12:52:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:59:43] a-team: I have workers at home (arrived ealier), going to be afk for ~1h.. please call me if anything is needed [12:59:51] ack elukey :) [13:25:31] hello joal, do we have it documented somewhere how to check the existence of a wiki in labsdb? [13:25:55] I'd like to include that in the ops duty item of adding a new wiki [13:25:57] nope fdans [13:26:13] fdans: magic script is the way ;) [13:26:22] oooo magic script! [13:26:36] joal: where's the magic script? [13:27:58] fdans: https://gist.github.com/jobar/dd915e2b47c8692871c0e254c54a09d8 [13:28:24] fdans: The queries against 'information_schema' are super long [13:28:40] hmm [13:29:23] joal: so why not have this script in the refinery script dir? seems pertinent [13:29:54] fdans: I used it when we started sqoop, then almost never [13:30:04] fdans: let's make it production ready if you want :) [13:30:50] right, I guess it has a pretty specific use case [13:47:03] heya teammm [13:49:12] (back) [13:49:30] hello mforns :) [13:49:40] hey elukey :] [13:51:36] !log kill mediacounts-load-coor, chown analytics:analytics /wmf/data/wmf/mediacounts, restart coordinator with user 'analytics' [13:51:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:52:42] hellooooo mforns [14:07:05] /wmf/data/wmf/mediacounts is taking ages to chmod sigh [14:07:14] chown [14:22:41] !log kill apis-coord and relaunch it with user analytics [14:22:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:24:18] fdans: o/ - there are two cassandra oozie coordinators in hue that are running with your username [14:24:25] are they meant to run or should we kill? [14:26:09] elukey: they are meant to run, but I should probably restart them with the hdfs user [14:26:49] I am currently moving all the hdfs users to the analytics one, it can be done if you want as part of it [14:28:06] (03PS1) 10Elukey: Move the oozie's apis-coord to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508330 (https://phabricator.wikimedia.org/T220971) [14:31:08] fdans: afaics they are only meant to push data to cassandra right? (so not writing to hdfs) [14:55:29] 10Analytics: API endpoint for mediacounts - https://phabricator.wikimedia.org/T207208 (10Nuria) >We need to be able to determine if people are interested in watching video viw Wikipedia. We all agree we need better media stats around video but https://tools.wmflabs.org/mediaviews/ does not expose data about vid... [14:59:20] (03CR) 10Nuria: Remove wikipedia-zero as program is over (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507933 (https://phabricator.wikimedia.org/T213770) (owner: 10Joal) [15:00:56] ping fdans , mforns , joal [15:01:23] ping fdans , mforns standddupppppp [15:03:50] nuria: sorry!! I had to drive my father in law to work and thought standup was an hour later, will be there for groskin [15:36:29] 10Analytics, 10Analytics-Kanban: Fix oozie banner_impression monthly job - https://phabricator.wikimedia.org/T222603 (10Milimetric) p:05Triage→03High [15:52:19] 10Analytics, 10Analytics-Kanban: Fix oozie banner_impression monthly job - https://phabricator.wikimedia.org/T222603 (10Milimetric) ping @DStrine do you have a timeline to move data collection for this schema to the EventLogging pipeline? We are fixing this job this time, but it's a duplicate of our standard... [15:52:56] 10Analytics, 10Analytics-Kanban: 15.wikipedia.org missclassified as a pageview, same for query.wikidata.org - https://phabricator.wikimedia.org/T222460 (10Milimetric) p:05Triage→03High a:05JAllemandou→03mforns [15:54:57] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Fix jobs after mediawiki-history refactor - https://phabricator.wikimedia.org/T222425 (10Milimetric) p:05Triage→03High [15:55:00] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Mandatory success_email_to parameter in mediawiki_history_check coordinator - https://phabricator.wikimedia.org/T222422 (10Milimetric) p:05Triage→03High [15:56:48] 10Analytics, 10Analytics-Kanban: Move the three sqoop jobs to oozie to ease administration and manual runs - https://phabricator.wikimedia.org/T222378 (10Milimetric) +1 to `&&` solution [15:57:03] 10Analytics, 10Analytics-Kanban: Move the three sqoop jobs to oozie to ease administration and manual runs - https://phabricator.wikimedia.org/T222378 (10Milimetric) p:05Normal→03High [15:58:38] 10Analytics, 10Analytics-Kanban: Move the three sqoop jobs to oozie to ease administration and manual runs - https://phabricator.wikimedia.org/T222378 (10Milimetric) a:03JAllemandou [16:06:10] 10Analytics, 10Analytics-Kanban: 15.wikipedia.org missclassified as a pageview, same for query.wikidata.org - https://phabricator.wikimedia.org/T222460 (10JAllemandou) here are the faulty lines: ` spark.sql("select uri_host, uri_path, uri_query from wmf.webrequest where webrequest_source = 'text' and year = 20... [16:06:52] mforns: I updated the pageview task wioth the faulty lines- I think the problem comes from their long uri_query [16:07:06] joal, thanks a lot, will look :] [16:07:18] Thanks you mforns (I had to do that last week ...) [16:07:22] sorry :S [16:07:34] no prob at all [16:14:45] 10Analytics, 10Analytics-Kanban: Fix oozie banner_impression monthly job - https://phabricator.wikimedia.org/T222603 (10JAllemandou) a:03JAllemandou [16:15:02] (03PS1) 10Joal: Fix oozie banner monthly job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508358 (https://phabricator.wikimedia.org/T222603) [16:28:17] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Fix oozie banner_impression monthly job - https://phabricator.wikimedia.org/T222603 (10DStrine) We will get back to event logging for CentralNotice soon but I don't have a timeline. We might have more information in a month or two. [16:29:00] 10Analytics, 10Analytics-Kanban: Mediawiki-history release - Snapshot 2019-05 - https://phabricator.wikimedia.org/T221825 (10JAllemandou) a:03JAllemandou [16:41:31] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10Marostegui) @elukey @Ottomata what do you guys want to do with this? [16:45:56] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10elukey) @Marostegui sorry I was under the impression that we'd have needed to wait for a feedback from Chris/Rob about how to proc... [16:49:42] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10Marostegui) @elukey sorry, I realised that I didn't sent the first sentence: "The errors corrected themselves and Icinga is now al... [16:50:01] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10Cmjohnson) I now have h/w log entries. I will need the server to be taken offline so I can relocate the DIMM and check to see if t... [16:50:44] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10Marostegui) @elukey can you coordinate with Chris? ^ [16:54:04] milimetric: as said in the wrong chan - Shall we deploy aqs with the new datasource tomorrow morning? [16:54:05] 10Analytics, 10Analytics-EventLogging, 10DBA, 10Operations, 10ops-eqiad: db1107 (eventlogging db master) possibly memory issues - https://phabricator.wikimedia.org/T222050 (10Marostegui) ` [18:50:55] marostegui i am confused over db1007...is there an issue or not an issue? There is a h/w l... [16:54:22] Will go for diner, back in a bit [16:56:54] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10MW-1.34-release, and 2 others: Remove deprecated EventLogging schema modules - https://phabricator.wikimedia.org/T221281 (10Krinkle) [17:26:36] * elukey off! [17:26:36] o/ [18:03:57] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Discovery, and 4 others: Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate - https://phabricator.wikimedia.org/T214080 (10Ottomata) @akosiaris I think https://wikitech.wikimedia.org/wiki/Kubernetes/He... [18:12:18] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Fundraising-Backlog, and 3 others: Fix EventLogging schemas that use array for items type - https://phabricator.wikimedia.org/T218617 (10Ottomata) @AndyRussG this should be do-able now, can you alter your schema? If you don't mind, I'm happy to... [18:14:28] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Fundraising-Backlog, and 3 others: Fix EventLogging schemas that use array for items type - https://phabricator.wikimedia.org/T218617 (10Ottomata) @Niharika If you are not the right person to ping about the TemplateWizard schema, could you point... [18:14:44] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Fundraising-Backlog, and 3 others: Fix EventLogging schemas that use array for items type - https://phabricator.wikimedia.org/T218617 (10Ottomata) [18:28:40] joal: yt? [18:32:19] need a hive eventlogging schema brain bounce [18:33:31] ottomata, if you can't wait for joal, I can try impersonate him [18:33:50] hm ok mforns ! bc? [18:34:01] yep! [18:51:08] 10Analytics: Tracking failures in your Matomo Analytics - https://phabricator.wikimedia.org/T222651 (10Dzahn) [19:12:19] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Fundraising-Backlog, and 3 others: Fix EventLogging schemas that use array for items type - https://phabricator.wikimedia.org/T218617 (10AndyRussG) @Ottomata Hi! I updated the [[ https://meta.wikimedia.org/wiki/Schema:CentralNoticeBannerHistory |... [19:28:19] 10Analytics, 10Analytics-Kanban, 10EventBus: Fix active EventLogging schemas that added backwards incompatable required fields. - https://phabricator.wikimedia.org/T222656 (10Ottomata) [19:28:30] mforns: oof ^ its nasty [19:30:36] 10Analytics, 10Analytics-Kanban, 10EventBus: Fix active EventLogging schemas that added backwards incompatable required fields. - https://phabricator.wikimedia.org/T222656 (10Ottomata) [19:31:28] 10Analytics, 10Analytics-Kanban, 10Discovery-Analysis, 10EventBus, 10Product-Analytics: Fix active EventLogging schemas that added backwards incompatable required fields. - https://phabricator.wikimedia.org/T222656 (10Ottomata) [20:01:21] ottomata, you mean it affects lots of schemas? [20:07:46] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Fundraising-Backlog, and 3 others: Fix EventLogging schemas that use array for items type - https://phabricator.wikimedia.org/T218617 (10Ottomata) Thank you! [20:09:39] (03CR) 10Nuria: [C: 03+1] browser-general-coord: move oozie job to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508307 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [20:09:44] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Change permissions for daily traffic anomaly reports on stat1007 - https://phabricator.wikimedia.org/T219546 (10ssingh) Hi @elukey. Thanks for your patience on this issue. I can confirm everything is working from my `~`. [20:10:01] (03CR) 10Nuria: [C: 03+1] Move oozie coords interlanguage/mediacounts to the new user analytics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508314 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [20:10:26] (03CR) 10Nuria: [C: 03+1] Move the oozie's apis-coord to the analytics user [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508330 (https://phabricator.wikimedia.org/T220971) (owner: 10Elukey) [20:10:42] mforns_brb: yeah and in different' revisions [20:10:54] its mostly the mobile wiki related schemas tho [20:15:12] (03CR) 10Nuria: [C: 03+1] Fix oozie banner monthly job (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/508358 (https://phabricator.wikimedia.org/T222603) (owner: 10Joal) [20:30:01] (03PS3) 10Fdans: Add 122 wikis that haven't been sqooped so far [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507355 (https://phabricator.wikimedia.org/T220456) [20:38:43] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) @akosiaris I think https://wikitech.wikimedia.org/wiki/Kubernetes/Helm... [20:44:02] (03CR) 10Nuria: "To confirm: we have tested these wikis appear on the lab hosts correct?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507355 (https://phabricator.wikimedia.org/T220456) (owner: 10Fdans) [21:03:16] (03CR) 10Fdans: "nuria: yep, tested with the magic script ™️" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507355 (https://phabricator.wikimedia.org/T220456) (owner: 10Fdans) [21:03:29] (03CR) 10Nuria: [C: 03+2] Add 122 wikis that haven't been sqooped so far [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507355 (https://phabricator.wikimedia.org/T220456) (owner: 10Fdans) [21:09:16] (03CR) 10Nuria: [V: 03+2 C: 03+2] Add 122 wikis that haven't been sqooped so far [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507355 (https://phabricator.wikimedia.org/T220456) (owner: 10Fdans) [21:28:40] (03PS1) 10Mforns: Filter out 15.wikipedia.org and query.wikidata.org from pageview definition [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/508418 (https://phabricator.wikimedia.org/T222460) [21:42:44] (03CR) 10Nuria: Filter out 15.wikipedia.org and query.wikidata.org from pageview definition (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/508418 (https://phabricator.wikimedia.org/T222460) (owner: 10Mforns) [21:43:22] nuria, thanks for CR [21:45:13] nuria, yes, have the same question [21:45:34] jo-seph said he thought the problem was related to very long URLs [21:45:43] mforns: ya, maybe worth seeing if there are 15.wikipedia pageviews on past months? [21:46:27] nuria, theoretically no, the code before my change already filters regular 15.wikipedia pageviews out, according to the tests I wrote [21:46:39] they pass with old code (just tested) [21:47:07] but yea, I will continue to troubleshoot this tomorrow [21:47:36] mforns: ya, i was wondering if 15.wikipedia.org is somewhow a new ocurrence [21:47:46] and there are no requests in past months [21:47:50] that would explain it