[00:35:04] <wikibugs>	 (03PS5) 10Smalyshev: Add tagger for Wikidata Query Service requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/364542 (https://phabricator.wikimedia.org/T169798)
[09:29:22] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Security, 10User-Addshore: Access rights for HDFS on stat100* for Sqoop tasks - https://phabricator.wikimedia.org/T170052#3434597 (10elukey) ``` elukey@stat1002:~$ hdfs dfs -ls /user/goransm Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 Found 1 items drwx------   -...
[09:43:41] * elukey errand + early lunch!
[09:46:09] <wikibugs>	 10Analytics: Code Review Needed: WMDE Summer Banner Campaign Analytics - stat1002 - https://phabricator.wikimedia.org/T170452#3431877 (10Tobi_WMDE_SW) @jan_dittrich are you familiar with R scripts and can have a quick look if they're fine?
[10:15:20] <wikibugs>	 10Analytics-Kanban, 10Wikimedia-Stream, 10Patch-For-Review: Decommission RCStream - https://phabricator.wikimedia.org/T170157#3434749 (10Xqt)
[10:19:07] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Reading-Web-Backlog, and 3 others: Duplicate events sent in Firefox after back button press - https://phabricator.wikimedia.org/T170018#3434776 (10phuedx) 05Open>03Resolved I can no longer reproduce this bug.  >>! In T170018#3424168, @phuedx wro...
[11:15:11] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10User-Elukey: Upgrade eventlogging servers to Jessie - https://phabricator.wikimedia.org/T114199#3434837 (10elukey)
[12:09:42] <wikibugs>	 10Analytics: Code Review Needed: WMDE Summer Banner Campaign Analytics - stat1002 - https://phabricator.wikimedia.org/T170452#3434931 (10Jan_Dittrich) @Tobias_Schumann_WMDE: I had a look and it looks fine to me, however I did not debug through it.
[12:51:37] <milimetric>	 hey morning yall
[12:53:01] <elukey>	 o/
[12:53:47] <elukey>	 milimetric: (whenever you have time) - do you remember if analytics-slave/store was set as read-only=true for a reason?
[12:54:57] <milimetric>	 hm.... like it's not accepting any more data from replication?
[12:55:21] <milimetric>	 or you mean just the research user doesn't have write permissions?
[12:55:36] <elukey>	 nono replication works but only because eventlogging_sync.sh runs as root :)
[12:55:51] <elukey>	 I tried to run the eventlogging_cleaner script on -slave and I got
[12:55:52] <elukey>	 pymysql.err.InternalError: (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement')
[12:56:00] <elukey>	 that makes sense, I am not running it as root
[12:59:50] <elukey>	 milimetric: the main issue is that db1046/47 (EL master and analytics-slave) are in HA using the m4-master domain
[13:00:06] <elukey>	 so say db1046 fails, db1047 takes over as m4-master
[13:00:07] <elukey>	 BUT
[13:00:13] <milimetric>	 yeah, I'm thinking, I don't remember why it's read-only but I would guess maybe performance?
[13:00:36] <elukey>	 since it is read-only, EL will fail to insert (at the moment)
[13:00:42] <milimetric>	 right
[13:00:51] <elukey>	 if readonly is false, then -slave would accept those inserts
[13:01:14] <elukey>	 the alternative, to avoid touching the current config, is running eventlogging_cleaner as root
[13:01:15] <milimetric>	 I do remember when Jaime did that
[13:01:20] <milimetric>	 but I don't remember why exactly
[13:01:34] <milimetric>	 yeah, I agree with you running as root is scary
[13:03:10] <elukey>	 but at the same time we have replication running as root
[13:05:00] <milimetric>	 I mean, you could have the script turn off read-only while executing and then turn it back on
[13:05:28] <milimetric>	 should probably check with the dbas either way
[13:05:42] <elukey>	 they are saying that running as root might be fine
[13:07:18] <milimetric>	 k
[13:08:00] <milimetric>	 btw, a-team, I'm having some really serious back pain today so if I scream at anyone or at the walls or something, don't worry, nothing's really wrong
[13:08:13] <elukey>	 :(
[13:08:16] <mforns>	 :/
[13:08:18] <milimetric>	 I may also just skip meetings, be generally unavailable as I writhe in pain
[13:08:21] <fdans>	 agh that sucks milimetric 
[13:08:28] <fdans>	 too many pushups?
[13:08:56] <milimetric>	 no, pushups are easy :)  It was hand planing 400 feet of cherry wood that did it
[13:12:07] <milimetric>	 elukey: I see jenkins +2-ed the wikistats deploy, wanna merge? :)
[13:12:46] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Manage application state with vuex - https://phabricator.wikimedia.org/T169371#3435175 (10Milimetric)
[13:12:51] <wikibugs>	 10Analytics-Tech-community-metrics: Add remaining KPIs to Overview once available in kibana - https://phabricator.wikimedia.org/T116572#3435176 (10Aklapper) 05stalled>03declined I'm declining this task as I don't consider the missing old KPIs particularly useful. One outcome / side effect of T160430 was the...
[13:12:59] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Manage application state with vuex - https://phabricator.wikimedia.org/T169371#3396211 (10Milimetric) a:05Milimetric>03fdans
[13:13:46] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3435189 (10Addshore)
[13:13:48] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10WMDE-Analytics-Engineering, 10Patch-For-Review, 10User-Addshore: Move statistics::wmde jobs from stat1002 -> stat1005 - https://phabricator.wikimedia.org/T170472#3432519 (10Addshore) 05Resolved>03Open Reopened until the above patch is merged
[13:14:06] <addshore>	 ottomata: ^^ for when you wake up, 1 tiny bit of adjustment needed for the wmde jobs!
[13:14:33] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Build Dashboard on top of dynamic data - https://phabricator.wikimedia.org/T167677#3435193 (10Milimetric) a:03fdans
[13:15:14] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Review by legal department of text on wikistats site - https://phabricator.wikimedia.org/T163229#3435195 (10Milimetric)
[13:16:40] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Backend  for wikistats 2.0 - https://phabricator.wikimedia.org/T156384#2973128 (10Milimetric) a:03Milimetric
[13:22:19] <elukey>	 milimetric: sure!
[13:23:07] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Backend  for wikistats 2.0 - https://phabricator.wikimedia.org/T156384#3435242 (10Milimetric) One version of the history schema simplified and loaded to test how Druid can work as a direct back-end for AQS:  https://pivot.wikimedia.org/#test-wks2-metrics
[13:23:13] <milimetric>	 thanks Luca
[13:26:15] * elukey runs puppet on thorium
[13:28:58] <elukey>	 milimetric: puppet run correctly, all good
[13:29:52] <elukey>	 milimetric: https://stats.wikimedia.org/v2 should be the target url?
[13:32:45] <elukey>	 ah apache might need to be tweaked for the new localtion /v2
[13:33:25] <elukey>	 Require all granted is for /srv/stats.wikimedia.org/htdocs
[13:34:14] <elukey>	 and the document root is DocumentRoot /srv/stats.wikimedia.org/htdocs
[13:47:13] <elukey>	 and now it works! https://stats.wikimedia.org/v2/#/all-projects
[13:47:29] <elukey>	 just need to move the symlink inside htdocs
[13:52:15] <fdans>	 oh jsonp not working milimetric ?
[13:52:45] <milimetric>	 fdans: it looks ok to me
[13:53:01] <fdans>	 ohhhh nooooo
[13:53:18] <fdans>	 ad blockers are blocking the jsonp requests
[13:53:26] <milimetric>	 yeah, that's common
[13:53:31] <milimetric>	 same thing happens to dashiki
[13:53:39] <elukey>	 milimetric: https://gerrit.wikimedia.org/r/#/c/365013/1/modules/statistics/manifests/sites/stats.pp
[13:54:01] <fdans>	 there's a workaround for that, I remember fixing that in the carto editor
[13:54:14] <milimetric>	 fdans: we might be able to do a straight request since wikimedia.org is whitelisted to do a straight query to other wikimedia.org, no?
[13:55:01] <milimetric>	 elukey: thanks, got it
[13:55:08] <fdans>	 hmmm different subdomains don't violate same-origin? I'm a bit rusty in all this
[13:55:21] <elukey>	 all right all working right?
[13:55:26] <elukey>	 (too many rights)
[13:55:33] <milimetric>	 they do, but I think the CORS policy allows *wikimedia.org
[13:55:38] <milimetric>	 not sure
[13:55:44] <milimetric>	 elukey: all working, thank you!
[13:56:06] <elukey>	 \o/
[13:56:26] <fdans>	 ah cool, then sure, just have to remove the jsonp property and add the cors header parameter
[14:05:07] <ottomata>	 addshore:  merged that change :)
[14:05:08] <ottomata>	 thanks
[14:05:42] <addshore>	 thanks ottomata !
[14:31:37] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Define, Document  (and test) Desktop and Mobile browser support for wikistats 2.0 - https://phabricator.wikimedia.org/T170457#3435660 (10mforns) a:03mforns
[14:35:13] <wikibugs>	 10Analytics, 10Mobile-Content-Service, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-iOS-App-Backlog: As an end-user I shouldn't see non-articles in the list of top articles - https://phabricator.wikimedia.org/T124082#3435677 (10Fjalapeno)
[14:35:36] <wikibugs>	 10Analytics, 10Mobile-Content-Service, 10Reading-Infrastructure-Team-Backlog, 10Wikipedia-iOS-App-Backlog: As an end-user I shouldn't see non-articles in the list of top articles - https://phabricator.wikimedia.org/T124082#1945028 (10Fjalapeno) @Nuria is this still a valid ticket?
[14:41:06] <wikibugs>	 (03CR) 10Mforns: "Queries are good! If we plan to run and get reports, we'll need to add some config in the config.yaml file, like:" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/364631 (https://phabricator.wikimedia.org/T170286) (owner: 10Nuria)
[14:42:53] <elukey>	 mforns: o/
[14:43:00] <mforns>	 elukey, hello!
[14:43:44] <mforns>	 sup?
[14:43:58] <fdans>	 mforns: wololoooo
[14:44:12] <mforns>	 fdans, sup too! :]
[14:44:39] <fdans>	 I'm going to get a maxibon, anyone want anything?
[14:44:45] <mforns>	 xD
[14:47:00] * elukey stares at fdans with a bit of rage
[14:47:14] <elukey>	 ottomata: while checking varnishkafka metrics I noticed that some hosts were missing
[14:47:22] <elukey>	 ottomata: and they all had this in the logs
[14:47:23] <elukey>	  WARNING  Failed to get lock. Is another instance of logster running?
[14:47:35] <elukey>	 removing the lock from /var/run/etc.. fixes the issue
[14:48:04] <fdans>	 https://usercontent.irccloud-cdn.com/file/rTAGNJ1X/Screen%20Shot%202017-07-13%20at%2016.47.16.png
[14:48:11] * fdans is melting faster than his maxibon
[14:50:15] <elukey>	 mforns: do I have the green light to execute the scriptwith 
[14:50:42] <mforns>	 elukey, forgot to commit the white-list changes, doing it right now. and you can CR and merge, one sec
[14:51:01] <elukey>	 sudo -u eventlogcleaner /usr/local/bin/eventlogging_cleaner --whitelist /etc/eventlogging/whitelist.tsv --dry-run --older-than 1000 --newer-than 1001
[14:51:10] <elukey>	 mforns: already done them
[14:51:20] <mforns>	 elukey, oh!
[14:51:28] <mforns>	 crap
[14:51:31] <elukey>	 :)
[14:51:47] <elukey>	 is 1000/1001 ok for you? it goes back to 2014
[14:51:48] <milimetric>	 ottomata: can we get druid upgraded soon? https://phabricator.wikimedia.org/T157977
[14:52:09] <mforns>	 elukey, sounds good
[14:52:14] <elukey>	 milimetric: IIRC we are blocked by java8 on the hadoop cluster
[14:52:18] <milimetric>	 oof
[14:52:33] <milimetric>	 yeah, this in turn blocks the wikistats backend
[14:52:45] <milimetric>	 because without monthly aggregations it's hard to tell how to design the system
[14:52:52] <milimetric>	 would be nice to know how it performs
[14:52:53] <elukey>	 waiiit a minute though, 0.9.x? 
[14:53:10] <milimetric>	 we were going to upgrade to 0.10, it's in the comments
[14:53:13] <elukey>	 don't recall if 0.10 was the problematic one
[14:53:46] <elukey>	 ah yes 0.10 sorry, scratch what I've said :)
[14:54:28] <elukey>	 milimetric: you need 0.9.2 right?
[14:54:39] <milimetric>	 ?
[14:54:39] <elukey>	 mforns: ok running!
[14:54:46] <milimetric>	 I thought we were upgrading to 0.10
[14:54:51] <mforns>	 O.O
[14:54:57] <milimetric>	 you're saying 0.10 is broken?
[14:55:26] <milimetric>	 (ok, it's ok, we can talk in standup - don't wanna bother you while running THE script :))
[14:56:41] <ottomata>	 milimetric:  in some version they made openjdk 8 a requirement
[14:56:43] <ottomata>	 not sure which one
[14:56:49] <milimetric>	 gotcha
[14:56:53] <mforns>	 elukey, can I create the task to remember white-listing MediaWikiInstallPingback as well? or you created it already as well?
[14:57:11] <elukey>	 ottomata: do we have 0.9.2 now? don't remember
[14:57:29] <mforns>	 too many as wells
[14:57:33] <ottomata>	 0.9.0
[14:57:36] <elukey>	 mforns: it is runningggggggggggggggggggggggg
[14:57:49] * elukey dances
[14:57:59] <mforns>	 music from "jaws" plays in the background
[14:58:43] <ottomata>	 :)
[15:01:13] <nuria_>	 ping elukey 
[15:01:33] <elukey>	 nuria_: sorry I was amused by eventlogging_cleaner
[15:01:36] <elukey>	 joining :D
[15:03:44] <elukey>	 mforns: for alter - 'start_ts': '20141016145526', 'end_ts': '20141017145526'
[15:03:56] <mforns>	 ok
[15:09:59] <elukey>	 mforns: the script it is still running, how it will take for the first run when we'll do it? A month? :D
[15:10:14] <mforns>	 mmmm
[15:10:26] <mforns>	 it should be stopping in short no?
[15:10:31] <mforns>	 I mean finishing
[15:11:06] <mforns>	 we can increment the batch_size
[15:12:07] <elukey>	 we could yes, maybe 10k
[15:12:16] <elukey>	 also db1047 is really slow
[15:14:29] <wikibugs>	 10Analytics-EventLogging, 10Analytics-Kanban: ChangesListHighlights events missing from MySQL starting 2017-07-11 - https://phabricator.wikimedia.org/T170486#3435958 (10Nuria)
[15:15:02] <wikibugs>	 10Analytics-EventLogging, 10Analytics-Kanban: ChangesListHighlights events missing from MySQL starting 2017-07-11 - https://phabricator.wikimedia.org/T170486#3432990 (10Nuria) a:03Ottomata
[15:15:48] <wikibugs>	 10Analytics: upgrade druid and pivot - https://phabricator.wikimedia.org/T157977#3435962 (10Ottomata)
[15:18:10] <wikibugs>	 10Analytics-EventLogging, 10Analytics-Kanban: ChangesListHighlights events missing from MySQL starting 2017-07-11 - https://phabricator.wikimedia.org/T170486#3435967 (10Ottomata) Since making the change yesterday, I see new events for this schema.  Roan, s'ok now?
[15:21:06] <mforns>	 elukey, for some schemas 1k is like couple minutes
[15:21:48] <mforns>	 so 1 day could be like 700 batches for a single table... so
[15:22:44] <mforns>	 I agree 10k is good
[15:24:36] <elukey>	 mforns: poor db1047 is updating the mobile tables
[15:24:51] <mforns>	 xD
[15:27:18] <wikibugs>	 10Analytics-Kanban: Upgrade Druid to 0.9.2 as a temporary measure - https://phabricator.wikimedia.org/T170590#3435988 (10Milimetric)
[15:37:49] <wikibugs>	 10Analytics-EventLogging, 10Analytics-Kanban: ChangesListHighlights events missing from MySQL starting 2017-07-11 - https://phabricator.wikimedia.org/T170486#3436028 (10Catrope) Yes, events are coming in again. The missing events didn't come back , though.  ``` mysql:research@s3-analytics-slave [log]> select c...
[15:38:15] <wikibugs>	 10Analytics-Kanban: Mediawiki History Druid indexing failed - https://phabricator.wikimedia.org/T170493#3436030 (10Milimetric) a:05Milimetric>03Nuria
[15:39:27] <elukey>	 mforns: we also need to keep an eye on https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1047&from=now-3h&to=now
[15:40:45] <mforns>	 elukey, k
[15:41:11] <wikibugs>	 10Analytics-EventLogging, 10Analytics-Kanban: ChangesListHighlights events missing from MySQL starting 2017-07-11 - https://phabricator.wikimedia.org/T170486#3436039 (10Nuria) @Catrope : we will be backfilling past events today, once that is done you should see the events that did not get persisted to the db....
[15:42:10] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 6 others: Record an  event every time a new content namespace page is created - https://phabricator.wikimedia.org/T150369#3436047 (10Nuria) We have fixed insertion issue and will be backfilling events today. @Ottomata...
[15:42:56] <nuria_>	 mforns: if you are so kind as to CR this oen we can merge too: https://gerrit.wikimedia.org/r/#/c/364518/
[15:43:21] <mforns>	 nuria_, looking
[15:44:41] <milimetric>	 sorry a-team, I'm taking the rest of the day off
[15:44:44] <nuria_>	 milimetric: now i am not sure if it is ingestion or thaer that joseph wanted to delete some of past data, will check regardless
[15:44:47] <milimetric>	 I tried to tough it out but it hurts so bad I can't think
[15:44:49] <mforns>	 take care milimetric 
[15:45:03] <mforns>	 :]
[15:45:04] <nuria_>	 milimetric: PLEASE
[15:45:05] <milimetric>	 I'll be out tomorrow too - Steph's surgery
[15:45:05] <fdans>	 milimetric: aw man, get well soon
[15:45:16] <nuria_>	 milimetric: sounds good
[15:45:19] <fdans>	 talk to you next week hopefully
[15:45:27] <milimetric>	 but I'll keep thinking about the back-end, yes, next week
[15:45:30] <milimetric>	 o/
[15:45:34] <nuria_>	 ciao
[15:45:48] <mforns>	 bye!
[15:50:30] <wikibugs>	 (03CR) 10Mforns: Tag annotation should reflect that a tagger can return several tags (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/364518 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)
[15:53:28] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban: Firewalls appear to be preventing spark executors from talking to spark driver on stat1005 - https://phabricator.wikimedia.org/T170496#3436111 (10Nuria)
[15:55:06] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban, 10WMDE-Analytics-Engineering, 10Patch-For-Review, 10User-Addshore: Move statistics::wmde jobs from stat1002 -> stat1005 - https://phabricator.wikimedia.org/T170472#3436114 (10Nuria)
[15:55:32] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move statistics::discovery jobs from stat1002 -> stat1005 - https://phabricator.wikimedia.org/T170471#3436115 (10Nuria)
[15:56:03] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Deploy Wikistats and analytics.wikimedia.org via SCAP - https://phabricator.wikimedia.org/T170429#3430944 (10Nuria) p:05Low>03Normal
[15:57:06] <wikibugs>	 10Analytics: Code Review Needed: WMDE Summer Banner Campaign Analytics - stat1002 - https://phabricator.wikimedia.org/T170452#3436119 (10Nuria) Could please describe the data that is being made public so we can look at the data, rather than having to look at the code that produces it?
[15:58:25] <wikibugs>	 10Analytics, 10CirrusSearch, 10Discovery, 10Discovery-Search, and 3 others: Define metrics for search result quality for the entity selector widget on wikidata. - https://phabricator.wikimedia.org/T170400#3429845 (10Nuria) Moving to radar as it doesn't seem there are any actionables for analytics.
[15:58:59] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban: Firewalls appear to be preventing spark executors from talking to spark driver on stat1005 - https://phabricator.wikimedia.org/T170496#3433298 (10elukey) ``` elukey@stat1005:~$ sudo systemctl stop ferm.service elukey@stat1005:~$ pyspark --master yarn Picked up JAVA_TOOL...
[16:01:49] <mforns>	 elukey, db1047 seems back to normal in grafana? (except for IO) did the script finish?
[16:03:23] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban, 10Security, 10User-Addshore: Access rights for HDFS on stat100* for Sqoop tasks - https://phabricator.wikimedia.org/T170052#3436134 (10Nuria)
[16:06:24] <elukey>	 mforns: nope, it is also executing alter tables from dbas :P
[16:06:37] <mforns>	 ah ok
[16:07:44] <wikibugs>	 10Analytics: Make Spark 2.1 easily available on new CDH5.10 cluster - https://phabricator.wikimedia.org/T158334#3436150 (10Nuria) All versions< 2.2 are affected by security issue, that will be also part of the value of upgrading
[16:08:15] <wikibugs>	 10Analytics: Make Spark 2.1 easily available on new CDH5.10 cluster - https://phabricator.wikimedia.org/T158334#3436151 (10Nuria) Ideally we will get this upgrade with the new cloudera distribution
[16:09:30] <wikibugs>	 10Analytics, 10Beta-Cluster-Infrastructure: deployment-kafka01 out of disk space - https://phabricator.wikimedia.org/T170523#3436168 (10greg) kafka -> #analytics  Analytics: please help diagnose/fix the beta cluster kafka host.
[16:09:32] <wikibugs>	 10Analytics-Kanban, 10EventBus, 10Easy, 10Services (watching): EventBus logs don't show up in logstash - https://phabricator.wikimedia.org/T153029#3436170 (10Nuria)
[16:10:32] <wikibugs>	 10Analytics, 10Beta-Cluster-Infrastructure: deployment-eventlogging03 out of disk space - https://phabricator.wikimedia.org/T170522#3436174 (10greg) eventlogging -> #analytics   Please help here as well. Thanks!
[16:11:58] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban: Firewalls appear to be preventing spark executors from talking to spark driver on stat1005 - https://phabricator.wikimedia.org/T170496#3436186 (10elukey) Ok so this seems to fail also on stat1002 right? I checked the iptables rules diff and I didn't find any.
[16:12:21] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban: Firewalls appear to be preventing spark executors from talking to spark driver on stat1005 - https://phabricator.wikimedia.org/T170496#3436190 (10elukey) p:05Triage>03Normal
[16:12:34] <wikibugs>	 10Analytics, 10Performance-Team: Explore NavigationTiming by faceted properties - https://phabricator.wikimedia.org/T166414#3436191 (10Nuria) This is agood test case for our eventlogging refine cc @Ottomata
[16:12:51] <wikibugs>	 10Analytics, 10Performance-Team: Explore NavigationTiming by faceted properties - EventLogging refine - https://phabricator.wikimedia.org/T166414#3436193 (10Nuria)
[16:14:24] <wikibugs>	 10Analytics, 10Analytics-Data-Quality, 10Datasets-Webstatscollector, 10Language-Team, and 5 others: Investigate anomalous views to pages with replacement characters - https://phabricator.wikimedia.org/T117945#3436205 (10Nuria)
[16:16:10] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban: Firewalls appear to be preventing spark executors from talking to spark driver on stat1005 - https://phabricator.wikimedia.org/T170496#3436220 (10EBernhardson) I thought it used to work on 1002, but indeed I'm not able to get it to start up now. I've been running things...
[16:17:58] <wikibugs>	 10Analytics: Pivot "MediaWiki history" data lake: Feature request for "Event Users" - https://phabricator.wikimedia.org/T161185#3436235 (10Nuria) @Neil_P._Quinn_WMF Sorry, it is still not clear what does "distinct unique counts" mean?  BTW, "active editors " is a metric computed on a time slot, we will provide t...
[16:24:33] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban: Firewalls appear to be preventing spark executors from talking to spark driver on stat1005 - https://phabricator.wikimedia.org/T170496#3436263 (10Ottomata) ! Iinnnnteresting, it doesn't work on 1002 either?  Weird.  Usually ACCEPTED means that it is just waiting for roo...
[16:27:32] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban: Firewalls appear to be preventing spark executors from talking to spark driver on stat1005 - https://phabricator.wikimedia.org/T170496#3436279 (10Ottomata) This happens to spark-shell too, so its not a pyspark specific thing.
[16:28:35] <nuria_>	 mforns: we did not talk about ops week AGAIN!
[16:28:37] <nuria_>	 argh
[16:28:43] <mforns>	 nuria_, oh
[16:28:57] <nuria_>	 let's d o it tomorrow, maybe fdans can start on rotation
[16:28:59] <mforns>	 I can do next week, and tomorrow we can talk
[16:29:05] <nuria_>	 I can too
[16:29:05] <mforns>	 k
[16:29:18] <mforns>	 but this week has been yours no?
[16:29:37] <mforns>	 I'd be next in line
[16:29:51] <mforns>	 so, theoretically next week should be me
[16:30:21] <mforns>	 oh, no, this week was Dan
[16:32:14] <wikibugs>	 (03CR) 10Nuria: "Looks good and straightforward, did you test code in 1002 with tagging UDF?" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/364542 (https://phabricator.wikimedia.org/T169798) (owner: 10Smalyshev)
[16:32:58] <wikibugs>	 (03CR) 10Smalyshev: "I tested it on stat1002, tags look fine" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/364542 (https://phabricator.wikimedia.org/T169798) (owner: 10Smalyshev)
[16:40:10] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Reading-Web-Backlog, and 3 others: Duplicate events sent in Firefox after back button press - https://phabricator.wikimedia.org/T170018#3436367 (10Nuria) Excellent, good work.
[16:45:41] <wikibugs>	 10Analytics, 10Services: Pageview data shoudl be served from restbase with a different license - https://phabricator.wikimedia.org/T170602#3436402 (10Nuria)
[16:47:04] <wikibugs>	 (03PS3) 10Nuria: Add "desktop by browser" tab to browser reports [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/364631 (https://phabricator.wikimedia.org/T170286)
[16:47:56] <wikibugs>	 (03PS4) 10Nuria: Add "desktop by browser" tab to browser reports [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/364631 (https://phabricator.wikimedia.org/T170286)
[16:48:42] <wikibugs>	 (03CR) 10Nuria: "Added config to calculate metric retroactively." [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/364631 (https://phabricator.wikimedia.org/T170286) (owner: 10Nuria)
[16:54:35] <wikibugs>	 (03PS3) 10Nuria: Tag annotation should reflect that a tagger can return several tags [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/364518 (https://phabricator.wikimedia.org/T164021)
[16:54:50] <wikibugs>	 (03CR) 10Nuria: Tag annotation should reflect that a tagger can return several tags (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/364518 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)
[16:55:30] <wikibugs>	 (03CR) 10Nuria: "Looks good, will like also @mforns to take a look" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/364542 (https://phabricator.wikimedia.org/T169798) (owner: 10Smalyshev)
[17:08:47] <wikibugs>	 (03CR) 10Mforns: [C: 031] "LGTM!" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/364542 (https://phabricator.wikimedia.org/T169798) (owner: 10Smalyshev)
[17:11:52] <wikibugs>	 10Analytics, 10Services: License for pageview data - https://phabricator.wikimedia.org/T170602#3436529 (10ZhouZ)
[17:12:21] <wikibugs>	 10Analytics, 10CirrusSearch, 10Discovery, 10MediaWiki-extensions-WikibaseRepository, and 2 others: Define metrics for search result quality for the entity selector widget on wikidata. - https://phabricator.wikimedia.org/T170400#3436532 (10debt) It doesn't look like there is anything actionable for the Sear...
[17:14:44] <wikibugs>	 (03CR) 10Mforns: Add "desktop by browser" tab to browser reports (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/364631 (https://phabricator.wikimedia.org/T170286) (owner: 10Nuria)
[17:17:37] <wikibugs>	 (03PS5) 10Nuria: Add "desktop by browser" tab to browser reports [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/364631 (https://phabricator.wikimedia.org/T170286)
[17:18:23] <wikibugs>	 (03CR) 10Nuria: Add "desktop by browser" tab to browser reports (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/364631 (https://phabricator.wikimedia.org/T170286) (owner: 10Nuria)
[17:18:28] <wikibugs>	 10Analytics, 10Services (watching): Add Accept header to web request Eventlogging - https://phabricator.wikimedia.org/T170606#3436564 (10Pchelolo)
[17:19:48] <wikibugs>	 10Analytics, 10Services, 10WMF-Legal: License for pageview data - https://phabricator.wikimedia.org/T170602#3436578 (10Framawiki)
[17:21:51] <wikibugs>	 10Analytics, 10Services, 10WMF-Legal: License for pageview data - https://phabricator.wikimedia.org/T170602#3436402 (10Pchelolo) There's a line in the top description specifying `Apache2`
[17:29:33] <wikibugs>	 10Analytics, 10Services (watching): Add Accept header to webrequest table - https://phabricator.wikimedia.org/T170606#3436655 (10Pchelolo)
[17:29:35] <wikibugs>	 10Analytics, 10Services (watching): Add Accept header to webrequest logs - https://phabricator.wikimedia.org/T170606#3436656 (10Ottomata)
[17:30:49] <wikibugs>	 10Analytics, 10Services (watching): Add Accept header to webrequest logs - https://phabricator.wikimedia.org/T170606#3436564 (10Ottomata) This shouldn't be difficult.  We'd need to configure varnishkafka to emit this header as part of the JSON webrequest log to Kafka, and alter the webrequest Hive table(s) to...
[17:31:49] <wikibugs>	 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Automatically sync mediawiki-identities/wikimedia-affiliations.json DB dump file with the data available on wikimedia.biterg.io - https://phabricator.wikimedia.org/T157898#3436678 (10Lcanasdiaz) There was an old method to sync databases...
[17:51:29] <nuria_>	 ottomata: added you to review to this one: https://gerrit.wikimedia.org/r/#/c/362310/
[17:51:54] * elukey off!!
[17:53:00] <nuria_>	 ottomata: i have not added a column before let me know if teh alter also goes somewhere
[17:53:06] <nuria_>	 ciao elukey 
[17:54:07] <wikibugs>	 (03CR) 10Ottomata: [C: 031] Adding "tags" column to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/362310 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)
[17:54:13] <ottomata>	 +1
[17:54:21] <ottomata>	 nuria_:  you got it right
[17:54:24] <ottomata>	 we run the alter manually
[17:54:37] <nuria_>	 ottomata: but we "merge" taht code right?
[17:54:39] <nuria_>	 *that
[17:57:01] <nuria_>	 ottomata: what apermits are needed to run teh alter can hdfs user do it?
[17:57:49] <nuria_>	 a-team: stopping eventlogging on beta to see if i can adress disk issues
[18:00:19] <neilpquinn>	 nuria_: did you ping me here yesterday? My client says I was mentioned but I can't find it in scrollback :)
[18:00:32] <nuria_>	 neilpquinn: yes, we have added edit count 
[18:00:43] <nuria_>	 neilpquinn: to mediawiki  edit data lake
[18:01:00] <neilpquinn>	 nuria_: ah ha, very exciting :)
[18:01:06] <nuria_>	 neilpquinn: we have vetted teh count a bit and before we close the task we like to ask you to vet it too
[18:01:25] <nuria_>	 neilpquinn: you can look (that is what we do) at known users (yours ,others)
[18:01:36] <nuria_>	 neilpquinn: have in mind it doesn't include flow edits
[18:01:49] <nuria_>	 neilpquinn: any other way you can think of vetting it is good too
[18:02:42] <neilpquinn>	 nuria_: okay, I can definitely do that! Might be the beginning of next week rather than this one though.
[18:03:03] <nuria_>	 neilpquinn: that is fine, will asign task to you for final sign off
[18:03:16] <nuria_>	 neilpquinn: docs are updated too but as always please modify if needed
[18:03:36] <wikibugs>	 10Analytics-Kanban, 10Contributors-Analysis, 10Patch-For-Review: Provide cumulative edit count in Data Lake edit data - https://phabricator.wikimedia.org/T161147#3436933 (10Neil_P._Quinn_WMF)
[18:03:45] <wikibugs>	 10Analytics-Kanban, 10Contributors-Analysis, 10Patch-For-Review: Provide cumulative edit count in Data Lake edit data - https://phabricator.wikimedia.org/T161147#3436935 (10Nuria) a:05JAllemandou>03Neil_P._Quinn_WMF
[18:03:59] <wikibugs>	 10Analytics-Kanban, 10Contributors-Analysis, 10Patch-For-Review: Provide cumulative edit count in Data Lake edit data - https://phabricator.wikimedia.org/T161147#3122928 (10Nuria) Assigning to neil to do final vetting.
[18:04:21] <ottomata>	 nuria_:  we merge the create ya, so we have an up to date schema committed
[18:04:29] <ottomata>	 this helps with testing, if we need to easily create test tables, etc.
[18:04:38] <ottomata>	 nuria_:  yes, you run with hdfs user
[18:04:53] <ottomata>	 nuria_: re EL backfill
[18:04:54] <nuria_>	 ottomata: ok, if you merge i will run alter, gasp!
[18:05:02] <nuria_>	 ottomata: yesssir
[18:05:08] <nuria_>	 ottomata: was just looking at your CR
[18:05:10] <wikibugs>	 (03CR) 10Ottomata: [C: 032] Adding "tags" column to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/362310 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)
[18:05:12] <wikibugs>	 (03CR) 10Ottomata: [V: 032 C: 032] Adding "tags" column to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/362310 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)
[18:05:23] <ottomata>	 nuria_: should https://gerrit.wikimedia.org/r/#/c/362233/1 be merged to?
[18:05:25] <ottomata>	 it is a parent
[18:05:47] <ottomata>	 nuria_:  re EL backfill, i'm not sure i can easily report number of missing events inserted
[18:06:08] <ottomata>	 i just inserted 10 events, i get the duplicate entry error, but it still logs those duplicates as 'inserted'
[18:06:12] <nuria_>	 ottomata: no, will abandon that patch, we reworked how the deletion works 
[18:06:21] <ottomata>	 k
[18:06:22] <nuria_>	 ottomata: ahahaha
[18:06:28] <ottomata>	 i see: Inserted 2 RelatedArticles_16352530 events in 0.006176 seconds
[18:06:29] <wikibugs>	 (03Abandoned) 10Nuria: Adding comments to data deletion script for druid storage [analytics/refinery] - 10https://gerrit.wikimedia.org/r/362233 (owner: 10Nuria)
[18:06:39] <ottomata>	 there are indeed 2 RelatedArticles_16352530 in the 10 lines i'm inserting
[18:06:49] <ottomata>	 but one of them has duplicate key error
[18:06:54] <nuria_>	 ottomata: i see, cause we are running db with "no-fail-if-inserting-duplicates"
[18:07:10] <nuria_>	 ottomata: ok, wasn't sure, cause we have chnaged that in the past
[18:07:37] <nuria_>	 ottomata: it makes sense cause due to batching some batch might include :"duplicated" events
[18:07:47] <nuria_>	 ottomata: but regradless we should insert them all
[18:08:42] <nuria_>	 *regardless
[18:09:42] <ottomata>	 yes, hm, i'm a little confused though
[18:09:43] <nuria_>	 ottomata: i see, calculating number of not present is not super easy cause is the number inserted -  duplicated key errors
[18:09:48] <ottomata>	 both of these 2 events
[18:09:49] <ottomata>	 are in mysql
[18:09:58] <ottomata>	 but only one duplicate key error is logged
[18:09:59] <nuria_>	 ottomata: whataha
[18:10:08] <nuria_>	 ottomata: ahhhh
[18:10:32] <nuria_>	 ottomata: i bet that is  alogging problem, nvm then
[18:10:44] <nuria_>	 ottomata: we will never be able to tell teh number of not inserted then
[18:10:59] <ottomata>	 hmmm, with replace=True
[18:10:59] <nuria_>	 ottomata: let me look at handler code again
[18:11:01] <ottomata>	 it logs both, as a warning
[18:11:28] <ottomata>	 so, i can do it that way, but i think that will re-write all of the past data to mysql with INSERT REPLACE
[18:11:31] <ottomata>	 won't bother the slaves
[18:11:36] <ottomata>	 but that'll make the master really busy
[18:11:47] <nuria_>	 ottomata: ya double duty
[18:11:49] <ottomata>	 yeah, nuria_ im' going to run with replace=False
[18:12:02] <ottomata>	 and then if we see roan's events we can assume we are good?
[18:12:09] <nuria_>	 ottomata: no, wait
[18:12:15] <nuria_>	 ottomata: with replace=false
[18:12:28] <nuria_>	 ottomata:  a batch with 10 events one of which alredy exists will fail
[18:12:55] <nuria_>	 ottomata: so we always have to do replce+true
[18:13:00] <nuria_>	 *replace=True
[18:13:09] <nuria_>	  or rather replace+false and batches of 1
[18:13:20] <nuria_>	 replace=False and batches of 1
[18:13:22] <ottomata>	 oh nuria_ this is gonna totally suck no matter what
[18:13:22] <nuria_>	 makes sense?
[18:13:25] <ottomata>	 this will insert on master
[18:13:29] <ottomata>	 but slaves only work with max timestamp or id
[18:13:32] <nuria_>	 ottomata: oh EVEN BETTER news
[18:13:48] <ottomata>	 nuria, not sure if that's how replace works, trying
[18:14:12] <nuria_>	 ottomata: we tested this a while back i *hope* i am remembering this right
[18:14:17] <ottomata>	 it is how it works.
[18:14:18] <ottomata>	 rats
[18:14:30] <ottomata>	 yup
[18:14:31] <ottomata>	 oof
[18:14:36] <ottomata>	 you are right
[18:14:38] <nuria_>	 ottomata: but timestamp is alredy on event right?
[18:14:52] <nuria_>	 ottomata: so we are reinserting timestamped events from yesterday
[18:14:55] <ottomata>	 ya but once replication has moved beyond, it doesn't look for old events
[18:15:01] <ottomata>	 we can insert these on master
[18:15:06] <nuria_>	 ottomata: alalalal
[18:15:08] <ottomata>	 but the eventlogging sync script won't ever look at them
[18:15:15] <nuria_>	 ottomata: and we will never see them on slaves
[18:15:23] <nuria_>	 ottomata: boy that replication is lovely #not
[18:15:30] <ottomata>	 yea
[18:16:20] <ottomata>	 nuria_:  not sure what to do here.
[18:16:30] <nuria_>	 ottomata: ok, then , let's document the % data loss (we can infer than from roan's schema) and note it on a ticket
[18:16:54] <ottomata>	 man this mysql eventlogging setup sucks so much
[18:17:03] <ottomata>	 we shoudl really make a serious project/goal to replace this
[18:18:05] <ottomata>	 nuria_:  i'm not sure we can easily infer that just from roan's schema
[18:18:26] <nuria_>	 ottomata: coudl not agree more
[18:19:16] <nuria_>	 *could
[18:19:27] <ottomata>	 nuria_:  i could script up something that is a little smarter for this backfill, and run it to insert in all 3 dbs
[18:19:42] <ottomata>	 iterate through files, check for uuid, if not present, insert in all 3 dbs
[18:20:19] <nuria_>	 ottomata: where will you check for uuid?
[18:20:24] <nuria_>	 ottomata: on db?
[18:20:28] <ottomata>	 ya
[18:20:51] <ottomata>	 oof i dunno
[18:23:27] <nuria_>	 ottomata: i know this is so offputting
[18:23:35] <nuria_>	 ottomata: we are always patching holes
[18:23:43] <nuria_>	 ottomata: in teh mysql setup
[18:23:45] <nuria_>	 *the
[18:28:46] <nuria_>	 ottomata: and remind me again why we could use regular replication?
[18:29:15] <ottomata>	 why we couldn't?  decision made before my time
[18:29:18] <ottomata>	 not sure
[18:32:04] <nuria_>	 ottomata: that might be what we need to fix 1st
[18:32:16] <ottomata>	 nuria_:  1st?  i mena for a bigger project
[18:32:18] <ottomata>	 scrap mysql
[18:32:20] <ottomata>	 its not worth it! :)
[18:32:28] <ottomata>	 milimetric: yt?
[18:34:01] <nuria_>	 fdans: how are you doing with the CI , do you know whoto ping?
[18:34:11] <nuria_>	 fdans: fine to say you are not there yet
[18:34:47] <fdans>	 nuria_: not there yet but would love to know who to ask
[18:35:21] <fdans>	 also I need to add a new test command for ci
[18:35:27] <nuria_>	 fdans: in the release channel 
[18:35:37] <fdans>	 that uses phantomjs + no watcher
[18:36:14] <nuria_>	 fdans: antoine (in your same TZ, france)
[18:36:27] <fdans>	 awesome
[18:36:32] <fdans>	 thank youuuu
[18:37:10] <wikibugs>	 (03PS3) 10Nuria: Add script to purge old mediawiki data snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355601 (https://phabricator.wikimedia.org/T162034) (owner: 10Mforns)
[18:39:17] <nuria_>	 ottomata: reality is that we are stuck with mysql for   a while
[18:39:48] <nuria_>	 fdans: his nick is hashar
[18:42:54] <ottomata>	 nuria_:  ya but we haven't ever seriously made a project to replace it
[18:42:58] <ottomata>	 i think if we made that a quarterly goal
[18:42:59] <ottomata>	 we could do
[18:43:01] <ottomata>	 it
[18:43:10] <ottomata>	 maybe 2 quarters
[18:43:36] <nuria_>	 ottomata: more like two quarters, but it would need to happen after kafka rebuild right? 
[18:44:54] <ottomata>	 ya
[18:45:39] <nuria_>	 ottomata: i have no objections, we are talking 6 months into teh future
[18:46:16] <nuria_>	 ottomata: ticket is here: https://phabricator.wikimedia.org/T159170
[18:51:38] <nuria_>	 ottomata: quaterly goal for q3 then, seems like next quarter it will be too much
[18:56:05] <ottomata>	 aye
[18:57:19] <wikibugs>	 (03PS5) 10Nuria: Adding "tags" column to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/362310 (https://phabricator.wikimedia.org/T164021)
[18:59:12] <wikibugs>	 (03CR) 10Nuria: "Fixed ancestry, can be merged" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/362310 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)
[18:59:34] <ottomata>	 nuria_:  i think a lot of events are missing.  i'm backfilling with a hacked up eventlogging mysql handler that only inserts if the uuid isn't present
[18:59:44] <ottomata>	 i'll run this same script from source files on all 3 mysql boxes
[18:59:46] <ottomata>	 bypassing replication
[18:59:50] <ottomata>	 little bit of manual replication :/
[18:59:53] <nuria_>	 ottomata: ok, thank you 
[19:00:00] <nuria_>	 ottomata: are batches size 1
[19:00:07] <ottomata>	 i'm also logging the table + uuid for every record
[19:00:10] <ottomata>	 no , 1000
[19:00:12] <nuria_>	 ottomata: or are you checking uuid before adding event to batch
[19:00:14] <ottomata>	 or 60 seconds
[19:00:18] <ottomata>	 checking uuid before adding to batch ya
[19:00:30] <nuria_>	 ottomata: k, ya, i though it would be one or the other
[19:00:35] <wikibugs>	 (03CR) 10Ottomata: [V: 032 C: 032] Adding "tags" column to webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/362310 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)
[19:00:52] <nuria_>	 ottomata: k, we should probably send note to analytics@
[19:01:25] <ottomata>	 ya will do
[19:01:45] <nuria_>	 ottomata: and we need an alarm on errors on mysql consumer
[19:02:07] <nuria_>	 ottomata: will file ticket
[19:02:36] <ottomata>	 aye, i think i can add this hacky feature as a flag to the mysql handler
[19:03:13] <ottomata>	 errrrr, kinda weird though
[19:03:13] <ottomata>	 hm
[19:03:19] <ottomata>	 makes the code schema specific hmmm
[19:03:44] <wikibugs>	 10Analytics, 10Analytics-EventLogging: Alarm on errors on /var/log/upstart/eventlogging* files - https://phabricator.wikimedia.org/T170620#3437262 (10Nuria)
[19:03:54] <ottomata>	 woudl be nice for backfilling like this in the future, wouldn't ned hacked up code
[19:04:01] <ottomata>	 even though the way im' doing it is pretty hacky
[19:04:35] <nuria_>	 ottomata: I think it will slow down the consumer a lot too
[19:04:43] <ottomata>	 ya, but just for backfilling
[19:04:44] <nuria_>	 ottomata: so i do not think we should add oit
[19:04:46] <ottomata>	 it'd normally be off
[19:04:48] <nuria_>	 *it
[19:05:30] <ottomata>	 hmmm, errrrr, there are a lot of events missing, i don't get how we weren't alamred about this
[19:05:40] <ottomata>	 we have alarms on mysql insertion rate
[19:05:56] <nuria_>	 ottomata: let's look at those for a sec
[19:06:08] <ottomata>	 yeah i'm looking at graph
[19:06:13] <ottomata>	 it is a little funky during the time
[19:06:17] <ottomata>	 because of the process flapping
[19:06:26] <ottomata>	 but total rate doesn't really drop
[19:07:14] <nuria_>	 ottomata: ya, just saw that
[19:07:27] <nuria_>	 ottomata: you can see the busrts
[19:07:36] <nuria_>	 ottomata: we need an error alarm
[19:09:20] <nuria_>	 ottomata: or a threshold one on teh 5 minute average which iwl increase if there is a restart
[19:10:09] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Implement pageview metric  in Wikistats UI - https://phabricator.wikimedia.org/T163817#3437298 (10Nuria)
[19:10:11] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: AQS Api works with DimensionalData - https://phabricator.wikimedia.org/T167681#3437297 (10Nuria) 05Open>03Resolved
[19:10:24] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Implement Wiki Selector Widget - https://phabricator.wikimedia.org/T167675#3437299 (10Nuria) 05Open>03Resolved
[19:10:26] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Implement pageview metric  in Wikistats UI - https://phabricator.wikimedia.org/T163817#3211246 (10Nuria)
[19:10:33] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Implement pageview metric  in Wikistats UI - https://phabricator.wikimedia.org/T163817#3211246 (10Nuria)
[19:10:35] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Deploy new Wikistats to stats.wikimedia.org/v2 - https://phabricator.wikimedia.org/T167684#3437302 (10Nuria) 05Open>03Resolved
[19:10:37] <ottomata>	 nuria_:  https://github.com/wikimedia/puppet/blob/production/modules/eventlogging/manifests/monitoring/graphite.pp#L58-L73
[19:10:45] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Implement pageview metric  in Wikistats UI - https://phabricator.wikimedia.org/T163817#3211246 (10Nuria)
[19:10:47] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Interface from Detail page to DimensionalData - https://phabricator.wikimedia.org/T167680#3437304 (10Nuria) 05Open>03Resolved
[19:11:12] <nuria_>	 ottomata: ya, the problem is that a restart has the opposite effect
[19:11:24] <nuria_>	 ottomata: rate increases (if measured in a short time period)
[19:11:41] <nuria_>	 ottomata: that alarm works for  aprocess that died
[19:11:55] <nuria_>	 ottomata: or a master that is not reachable
[19:12:24] <nuria_>	 ottomata: does that make sense?
[19:12:49] <nuria_>	 ottomata: continuous restarts produce ( on 5 minutes timespan)  up-spikes
[19:13:54] <RoanKattouw>	 11:18:06 <ottomata> nuria_:  i'm not sure we can easily infer that just from roan's schema
[19:14:07] <nuria_>	 RoanKattouw: yessir
[19:14:10] <RoanKattouw>	 You can't infer it from *that* schema because it's too low-traffic, but I have a higher-traffic schema you could look at
[19:14:15] <RoanKattouw>	 ChangesListFilters
[19:14:28] <ottomata>	 ya
[19:14:33] <nuria_>	 RoanKattouw: we are backfilling it all for the last two days
[19:14:37] <RoanKattouw>	 Ooh sweet
[19:14:40] <nuria_>	 ooh the joy
[19:14:41] <ottomata>	 ya RoanKattouw i think there was a LOT of data that was not inserted
[19:14:47] <ottomata>	 i'm surprised no one noticed before you
[19:14:59] <RoanKattouw>	 I'm also going to look at all-events.log and compare daily counts going back to the start of that schema ~1 month ago to see if there were any other loss events
[19:15:01] <ottomata>	 i'll no how many events total by the end of this
[19:15:06] <nuria_>	 RoanKattouw: we now understand why our alarm did not go off
[19:15:18] <RoanKattouw>	 Yeah I have to say I later wondered about monitoring myself
[19:16:00] <RoanKattouw>	 And was thinking like "maybe events can be inserted periodically to check if they show up" but I now realize that the # of events inserted at all was probably worryingly low on its own
[19:16:01] <ottomata>	 most data did get inserted, so the total insertion rate didn't drop enough for us to get an alarm
[19:16:07] <RoanKattouw>	 So we do have monitoring but it broke?
[19:16:09] <RoanKattouw>	 Aha
[19:16:19] <RoanKattouw>	 Right, even my schema got a little bit of data during that time
[19:16:23] <nuria_>	 RoanKattouw: ya, see: https://grafana.wikimedia.org/dashboard/db/eventlogging?orgId=1&from=1499368569836&to=1499973069836&panelId=12&fullscreen&var-topic=All
[19:16:40] <nuria_>	 RoanKattouw: issue does not make insertion rate drop (which is what we alarm on) 
[19:16:46] <ottomata>	 ya, we audit this type of thing for webrequest data in hadoop
[19:16:50] <ottomata>	 by using sequence numbers
[19:16:57] <ottomata>	 we don't really have that for eventlogging
[19:16:59] <nuria_>	 RoanKattouw: but rather increase if you measure it in a short timespan
[19:17:17] <nuria_>	 RoanKattouw: rate was fine, thus no alarm
[19:17:31] <RoanKattouw>	 It made the rate *increase*?!
[19:17:40] <RoanKattouw>	 More insertions of smaller batches?
[19:17:56] <nuria_>	 RoanKattouw: more frequent insertions of batches of equal size
[19:18:22] <nuria_>	 "overall"
[19:18:36] <nuria_>	 RoanKattouw: batch size is not dynamic
[19:18:45] <nuria_>	 RoanKattouw: it is configured
[19:19:56] <nuria_>	 RoanKattouw: at least that is my theory as to teh bursts
[19:19:58] <nuria_>	 *the
[19:24:45] <nuria_>	 RoanKattouw: that i'm sure it convinced ottomata 
[19:26:36] <nuria_>	 #not
[19:26:49] <ottomata>	 actually, nuria_ Hmmmm about the replication thing...for tables that use id instead of timestamp for repliation
[19:26:51] <ottomata>	 it will work
[19:27:01] <ottomata>	 since these backfilled records will get more recent auto inc idds
[19:27:02] <ottomata>	 ids
[19:27:07] <nuria_>	 ottomata: ahhh
[19:27:25] <nuria_>	 ottomata: is that the majority?
[19:28:56] <ottomata>	 not totally sure
[19:33:11] <ottomata>	 RoanKattouw: at least a few of these backfilled events are def yours! 
[19:33:11] <ottomata>	 ChangesListHighlights_16484288 1387c02b4b625b6ea93fd0811978785b 1499844841 missing, inserting now
[19:34:28] <RoanKattouw>	 Excellent!
[19:34:47] <RoanKattouw>	 After lunch I will compare all-events.log to MySQL to see if there were any other loss events affecting this table
[19:34:58] <RoanKattouw>	 When I do that I will also find the restored events
[19:36:21] <wikibugs>	 (03PS1) 10Nuria: Corrected quotes in create syntax for webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/365089 (https://phabricator.wikimedia.org/T164021)
[19:36:45] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10MediaWiki-Vagrant, 10Services (watching): Vagrant git-update error for event logging - https://phabricator.wikimedia.org/T161935#3437415 (10Pchelolo) 05Open>03Invalid Ok, seems like something was corrupted in my vagrant setup. After deleting the virtualenv and r...
[19:40:40] <wikibugs>	 (03CR) 10Nuria: "Should have tested syntax on my own db before , as i would have caught  this issue earlier." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/365089 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)
[19:41:32] <nuria_>	 RoanKattouw: k, things need to get to masters (where you query) 
[19:41:39] <nuria_>	 RoanKattouw: sorry, slaves
[19:41:51] <RoanKattouw>	 Yeah no worries, I need to obtain food and eat it first :)
[19:45:41] <nuria_>	 fdans: FYI that wikistats docs for administration and such are in wikitech: https://wikitech.wikimedia.org/wiki/Analytics/Wikistats2.0
[19:45:48] <nuria_>	 fdans: not in mediawiki
[19:46:07] <nuria_>	 fdans: in mediawiki we have consultation docs and others geared towards the community
[19:50:20] <wikibugs>	 (03CR) 10Ottomata: [V: 032 C: 032] Corrected quotes in create syntax for webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/365089 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)
[19:50:37] <ottomata>	 this is also taking a while
[19:50:52] <nuria_>	 ottomata: puf, yaeh
[19:50:55] <nuria_>	 *yeah
[19:50:56] <ottomata>	 i'm running 3 processes (since there were 3 files), but it has to check every single record
[19:50:58] <nuria_>	 yeah
[20:01:50] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban: Firewalls appear to be preventing spark executors from talking to spark driver on stat1005 - https://phabricator.wikimedia.org/T170496#3437485 (10Ottomata) Oh!  In my puppet refactoring, I applied base firewall to stat1002 and stat1005.  It's never been applied to stat1...
[20:05:39] <wikibugs>	 10Analytics: Code Review Needed: WMDE Summer Banner Campaign Analytics - stat1002 - https://phabricator.wikimedia.org/T170452#3437514 (10GoranSMilovanovic) @Nuria The data are found in /a/published-datasets/wmdecampaigns/sbc2017/ on stat1002  sbc2017_PROD_BannerImpressions.R extracts only the uri_query field fro...
[20:06:56] <wikibugs>	 10Analytics, 10EventBus, 10Services (watching): Treat 'null' as 'undefined' when verifying the event schemas - https://phabricator.wikimedia.org/T170629#3437516 (10Pchelolo)
[20:08:00] <wikibugs>	 10Analytics, 10EventBus, 10Services (watching): Treat 'null' as 'undefined' when verifying the event schemas - https://phabricator.wikimedia.org/T170629#3437531 (10Ottomata) I think we can allow nulls, but not unless we explicitly want to.  I think it should be done in the schema explicitly if we do, right?
[20:11:24] <wikibugs>	 10Analytics: Code Review Needed: WMDE Summer Banner Campaign Analytics - stat1002 - https://phabricator.wikimedia.org/T170452#3437534 (10Nuria) @GoranSMilovanovic   Would you be so kind as to check with us whether data can be public before it is public?  Let's please stop these jobs from producing data until we...
[20:12:44] <wikibugs>	 10Analytics, 10EventBus, 10Services (watching): Treat 'null' as 'undefined' when verifying the event schemas - https://phabricator.wikimedia.org/T170629#3437536 (10Pchelolo) @Ottomata I was more wondering if we can just treat null as undefined generically and whether that's a good idea. So that if the field...
[20:17:52] <wikibugs>	 10Analytics, 10EventBus, 10Services (watching): Treat 'null' as 'undefined' when verifying the event schemas - https://phabricator.wikimedia.org/T170629#3437547 (10Ottomata) Hm, my first thought is that's a bad idea, right?  Since null can be used with a meaning, rather than just undefined, it seems like it...
[20:20:45] <wikibugs>	 10Analytics, 10EventBus, 10Services (watching): Treat 'null' as 'undefined' when verifying the event schemas - https://phabricator.wikimedia.org/T170629#3437550 (10Pchelolo) 05Open>03declined We don't really have any place where we use 'null' meaningfully yet, but I do agree it might be the case in the f...
[20:22:59] <wikibugs>	 10Analytics: Code Review Needed: WMDE Summer Banner Campaign Analytics - stat1002 - https://phabricator.wikimedia.org/T170452#3437555 (10GoranSMilovanovic) @Nuria   Unfortunately, the code had to be developed in a rapid manner, for a campaign that we have started running on 07/11/2017, so there was not time to a...
[20:25:31] <wikibugs>	 10Analytics: Code Review Needed: WMDE Summer Banner Campaign Analytics - stat1002 - https://phabricator.wikimedia.org/T170452#3437557 (10GoranSMilovanovic) 05Open>03Resolved a:03GoranSMilovanovic
[22:43:37] <nuria_>	 ottomata: still there?
[23:46:29] <wikibugs>	 10Analytics: Code Review Needed: WMDE Summer Banner Campaign Analytics - stat1002 - https://phabricator.wikimedia.org/T170452#3438057 (10Nuria) @GoranSMilovanovic Normally when we made data public is for the word to consume and we document datasets, announce them and make them available (after having stablished...