[00:43:20] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2307604 (kaldari) I've made a request for interim funding for using Google's Search API.... [03:50:11] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, and 2 others: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2307724 (Beetstra) @Sadads, @Kaldari - if this is supposed to help the anti-spam efforts, this should... [06:01:09] (PS1) Mobrovac: Scap: execute checks after the restart_service stage [analytics/aqs/deploy] - https://gerrit.wikimedia.org/r/289594 (https://phabricator.wikimedia.org/T135609) [06:03:41] elukey: joal: ^ [06:17:01] mobrovac: will review it! Thanks [06:17:43] also morning ;) [06:18:24] buongiorno1 [06:18:25] ! [07:01:57] Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: Add kartik to analytics-privatedata-users group - https://phabricator.wikimedia.org/T135704#2307853 (KartikMistry) [07:10:12] Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: Add kartik to analytics-privatedata-users group - https://phabricator.wikimedia.org/T135704#2307872 (jcrespo) I suggested this request so their queries can be done on analytics s... [09:29:42] joal: o/ [09:32:33] do want to see the latest on aqs? [09:32:39] or hear better [09:33:38] we are running cassandra 2.1.12 atm, restbase 2.1.13 buuut since in the apt repo we have 2.1.14 try to guess what ended up on aqs100[456] ? [09:36:03] (now I get what urandom was telling me yesterday about https://phabricator.wikimedia.org/T95253) [09:46:17] Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: Add kartik to analytics-privatedata-users group - https://phabricator.wikimedia.org/T135704#2308172 (Joe) p:Triage>Normal [09:47:25] Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: Add kartik to analytics-privatedata-users group - https://phabricator.wikimedia.org/T135704#2307853 (Joe) We will need manager's approval for this request for Kartik. In the mea... [10:12:41] ok joal I *think* that doing something like "stop cassandra instance, downgrading cassandra, starting cassandra instance" might work [10:13:16] if you have ever watched "Slevin" this is the kansas city move [10:17:46] hi joal, elukey and team, I have to go to the airport and fetch a rented car this morning, I will be working on east coast hours today [10:18:17] mforns: okkkkk [10:18:44] see ya :] [10:29:04] Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: Add kartik to analytics-privatedata-users group - https://phabricator.wikimedia.org/T135704#2308272 (Arrbee) This is an approved request for Kartik. Thanks. [10:39:56] * elukey lunch, brb [11:24:48] urandom: nice!! https://apachebigdata2016.sched.org/speaker/john.eric.evans [11:32:11] joal: we'd need to restart Hue and Oozie :D [12:24:35] joal: aqs100[456] downgraded to 2.1.13, so you can start adding data :P [12:28:33] !log restarted hue on analytics1027 for security upgrades [12:29:02] nice elukey! [12:29:06] thnx for the downgrade [12:30:50] mobrovac: it was easy with an empty cluster :P [12:31:01] hehe [12:31:10] mobrovac: what is the plan for aqs100[123]? It is still running on 2.1.12 [12:32:10] !log suspended all the oozie bundles as prep step for oozie's restart (security upgrades) [12:32:12] not sure what's the smartest move tbh elukey [12:32:21] given the soon-ish move to 2.2.6 [12:33:05] either way, one thing you should know is that aqs100[123] and aqs100[456] ought to have the same cass version once you start joining the clusters and replacing them [12:34:15] mobrovac: yep.. and I'd prefer to do the migration before 2.2.6 :D [12:34:51] so I think that we should move aqs100[123] to 2.1.14, meanwhile testing aqs100[456] and eventually joining the two [12:34:58] final step 2.2.6 only on aqs100[456] [12:35:21] elukey: s/2.1.14/2.1.13/ [12:36:27] yes sorry, [12:37:18] :P [12:37:24] but yeah, that sounds sensible [12:37:36] better than upgrading 6 nodes to 2.2.6 only to get rid of 3 of them [12:44:57] !log restarted oozie on analytics1003 for security upgrades [13:25:17] Analytics, ArticlePlaceholder, Pageviews-API, Wikidata: Track pageviews of ArticlePlaceholders - https://phabricator.wikimedia.org/T132223#2308862 (Lucie) [13:25:24] Analytics, ArticlePlaceholder, Pageviews-API, Wikidata: Track pageviews of ArticlePlaceholders - https://phabricator.wikimedia.org/T132223#2308864 (Lucie) a:Addshore>None [13:30:14] Analytics-Kanban, Operations, ops-eqiad, Patch-For-Review: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2308897 (elukey) I followed @fgiunchedi's advice and had a chat with @ema about this. His code updates initramfs only after the first time that puppet runs, meanwhi... [13:34:34] Hi elukey [13:34:43] Hi elukeysorry had trouble connecting before [13:34:54] Thanks for handling both cassandra and oozie elukey ! [13:35:36] joal: o/ [13:35:42] elukey: Heyy :) [13:35:57] Had you paused every bundle before restarting A? [13:35:58] soo finally you can put stuff on cassandra :P [13:36:07] elukey: awesome ! [13:36:20] Will start playing this afternoon [13:36:22] yes all of them, waited also for yarn -> running [13:36:30] everything looked good [13:36:36] then restarted and checked for a while [13:36:38] nothing weird came up [13:37:03] elukey: so that you know, only load jobs are really needed (webrequest and mediawiki) [13:37:21] When those ones are paused, everything else naturally stops [13:37:33] But pausing everything works as well :) [13:40:04] joal: yess I remember it because you patiently explained to me a lot of times, buuuut I wanted to be extra careful :P [13:41:11] np elukey, better more than less :) [13:42:32] elukey : So I can try connecting to cassandra on aqs1004 on port 9042? [13:43:01] elukey@aqs1004:~$ sudo netstat -nlpt [13:43:04] tcp6 0 0 10.64.0.127:9042 :::* LISTEN 2240/java [13:43:07] tcp6 0 0 10.64.0.126:9042 :::* LISTEN 2239/java [13:43:49] so you should use aqs1004-{ab}.eqiad.wmnet [13:44:13] same thing for the other two guys 1005/6 [13:45:01] joal --^ [13:47:40] elukey: Ooooh, they have different hostnames? [13:47:45] Sweet [13:50:12] joal: I've also updated https://wikitech.wikimedia.org/wiki/Analytics/AQS [13:51:32] That's great elukey [13:52:06] elukey: here is my plan: currently there is an existing keyspace for per-article, created by restbase [13:52:14] This keyspace uses DTCS [13:53:34] My plan is to create another keyspace using the same config; except for compation, and test to load one day of data, and see, then multiple days etc [13:54:00] First monitoring load/compression only (no read yet) [13:56:19] +1 [13:56:37] joal: let me know if I can help, really curious [13:57:19] elukey: I'll ping you when starting, to show you [13:57:38] elukey: currently doing some work on pageview backfilling [13:57:43] sure! [13:59:42] elukey: Actually will try to alter the existing table if you agree [14:00:15] elukey: mwarf ... wondering ... [14:02:33] :D [14:03:51] Analytics-Tech-community-metrics, Phabricator-Upstream, Upstream: List of Phabricator users - https://phabricator.wikimedia.org/T37508#400343 (Aklapper) >>! In T37508#2302594, @jayvdb wrote: > Was the "People" a WMF requested feature? Not that I know. > It seems to have existed long before WMF adop... [14:05:44] elukey: Let's be safe, try a new keysapce :) [14:07:16] joal: well the keyspaces are empty atm, and we can re-create the cluster very easity [14:07:19] *easily [14:07:38] so if you prefer to play the card of altering keyspace, you can try :) [14:07:48] elukey: yes, but I want to be sure restbase won't change the compaction strategy in the middle of us testing ;) [14:07:59] So trying with a new one :) [14:09:43] New keyspace created [14:09:57] Now, a loading job :) [14:12:18] gooooood [14:12:55] joal: is it possible to run a node with DTCS and another one with LC, or is it a per-keyspace property? I don't remember [14:13:14] elukey: per-table actually, not keyspace [14:14:02] joal: table == section of the keyspace ending up in node X ? (super ignorant about this) [14:14:36] elukey: table = abstract view of the stuff you store in a keyspace [14:15:17] elukey: in a keyspace, you can have multiple tables, each having a data schema, and various config options (compaction, compression etc) [14:15:38] ahhhh [14:16:19] so what happens is we choose LC over DTCS? We will need to join one new instance at the time to the pre-existing cluster that has tables already with DTCS [14:16:37] (at least, the per article one) [14:16:46] maybe we already discussed it but I forgot sorry :( [14:17:39] elukey: IIRC urandom said you could set compaction strategy per node, but I'm not expert enough to know how :) [14:17:48] I only know the easy way ;) [14:19:21] joal: all right I'll ask him, this is kinda important [14:19:29] it is indeed [14:19:37] not for now, but for after [14:19:48] yep [14:19:59] joal: completely unrelated news, https://www.youtube.com/user/Kurzgesagt [14:20:17] I think that you'll like it, those folks are amazing [14:21:25] elukey: Thanks a million ! Not a good time now, too many things to do, but will definitely keep it for a quiter time :) [14:35:22] Analytics, Revision-Slider, TCB-Team, WMDE-Analytics-Engineering, and 2 others: Data need: User Behaviour when comparing article revisions - https://phabricator.wikimedia.org/T134861#2309205 (Addshore) [14:49:41] urandom: goooood morning :) [14:50:21] whenever you have time I have a couple of questions [14:50:41] elukey: morning! i have time! [14:51:43] thanks! [14:52:13] I've read your comment about downgrading aqs100[123] to 2.1.12 but I'd need some guidance about how to prepare the work [14:52:29] sorry, upgrading to 2.1.13 [14:52:45] I have too versions in my head atm so I confuse them :P [14:52:51] yeah :) [14:53:22] so, 2.1.13 has been thoroughly tested here, under the config you are using (basically you are using the same config as us) [14:53:48] and the upgrade was very straightforward [14:54:44] tl;dr you should be able to just upgrade the package, and bounce the machine [14:54:54] ah nice [14:55:12] in a rolling fashion, of course [14:55:19] yeah not all in once P [14:55:20] :P [14:55:28] but otherwise nothing special should be needed [14:55:35] *this time* :) [14:55:36] bounce the machine == simply restart cassandra right? [14:55:40] yeah [14:55:43] all right [14:56:12] joal: --^ [14:56:22] elukey: you could do one first, use it as a canary [14:56:41] but make sure to complete the upgrade before attempting any bootstraps and/or decommissions [14:57:01] urandom: all right, so I can keep multiple versions as long as I don't add/remove nodes [14:57:10] add/remove == bootstrap/decommission [14:57:39] well, best-practice for a cluster of machines, would be to run mixed versions as part of a transition from one version to another [14:57:59] but not for any longer than is necessary to complete that transition [14:58:22] so your new nodes are not a part of the production cluster yet, i assume [14:58:25] ? [14:58:31] last i checked, anyway [14:58:36] so that is fine [14:59:28] but once you start upgrading your prod cluster, you should plan to finish that upgrade in whatever time makes sense [14:59:45] like, you could upgrade one now, and come back and do the rest on monday, that would probably be OK [15:00:13] but i wouldn't leave it open-ended :) [15:00:14] urandom: correct 100[456] are only joal's playground for the moment :P [15:01:42] urandom: second question! We'd like to test LC instead of DTCS, and I was thinking if it is possible to assign a specific compaction setting for a keyspace/table to a node, in order for example to have a mixed config during the migration. [15:01:56] you have already answered this question I know, but I don't recall the answer :) [15:03:48] elukey: yes, it can be done using a jmx op, the setting is ephemeral though; it'll revert if the node is restarted [15:04:17] elukey: where you looking for this? http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-1-live-traffic-sampling [15:04:35] joining a node without joining it? so-called write-survey mode? [15:05:35] elukey: a test of LC should be done under import [15:05:47] that is what will make-or-break it [15:06:04] you could do that now, while you have the new machines setup as a test cluster [15:06:59] urandom: what we want to do is load real data to aqs100[456] setting LC straight away and check how it goes [15:07:06] as alternative to write survey [15:07:40] ok, can't you just set the compaction normally... oh damn, restbase changes it back... [15:07:44] * urandom sighs [15:08:09] is that the problem? if you alter the keyspace restbase alters it back? [15:08:55] urandom: nono I was wondering what to do if we like LC, because I'll need to eventually join the new nodes/instances to a cluster with pre-existing settings [15:09:11] oh i see [15:09:39] thinking out loud here... [15:09:57] one option would be to bootstrap the new nodes, decomm the old, and then alter the keyspace [15:10:25] that way you have all SSDs, and the IO to handle a recompaction [15:11:32] the inverse would be to alter, and then start bootstrapping the new nodes, but that puts the existing nodes to work recompacting, which might not be awesome if they are already struggling under high IO [15:12:19] i suppose you could use jmx to poke one of the existing nodes into LC long enough to guage the impact, maybe during an off-peak period [15:13:13] i think what you are suggesting, is to bootstrap a new node, for the compaction strategy locally, and then lather-rinse-repeat for each new node/instance [15:13:25] s/for the compaction/force the compaction/ [15:13:36] yep... [15:13:47] a lot of homework to do :P [15:14:11] which might work... but any restart will revert it back to DTCS, and cause it to start recompaction again [15:14:36] seems a little janky [15:14:57] I think at this point that migration and alter AFTER on ssd might be the best and cleanest option, having data from our current testing [15:15:01] maybe there is a way to automate this [15:16:06] mobrovac: are you here? [15:17:13] elukey: if we could disable compaction on startup, then automate the change in compaction strategy, and reenabling of compaction, as part of the startup sequence, then maybe this would work [15:17:54] otherwise i'd be worried that every routine restart would generate problem [15:19:21] heh, i wonder if we could set concurrent compactors to 0 in the config [15:19:36] urandom: might be best to try the path that we know could work, like altering the compaction settings after the migration.. we already done if moving to DTCS for aqs right? [15:20:41] elukey: well, if you *know* you intend to move to LCS immediately after, then it might be worth thinking about ways to avoid doubling handling all of that compaction work [15:20:59] true true [15:21:28] this would probably be easy to test, setting concurrent_compactors to 0 [15:22:32] if that disables compaction, then we could script a startup sequence that locally altered the strategy, and then set concurrent_compactors to it's usual setting via nodetool [15:22:51] and use this temporarily until your migration was complete [15:23:02] then alter, and undo the startup hacks [15:23:14] worth thinking about [15:23:32] so basically trickying the new hosts with a different statup sequence until the old ones are deprecated [15:23:43] yeah [15:23:57] mmmmm starting to like the idea [15:24:05] I'll add it as a test to perform [15:24:29] maybe wait an see how LCS works out for you, before investing too much time into it [15:24:37] yep yep [15:25:25] last question I promise: where did you get cassandra_2.1.13_all.deb? Is it from the debian stable repo right? Just wanted to have a rollback to 2.1.12 if needed [15:25:41] urandom: thinking aloud as well on the topic: Couldn't we use the jmx op trick and monitor no restart while bootstrapping? [15:26:46] (PS5) Nuria: Initial content of analytics.wikimedia.org [analytics/analytics.wikimedia.org] - https://gerrit.wikimedia.org/r/289062 (https://phabricator.wikimedia.org/T134506) [15:27:03] joal: in the event a restart was necessary, you'd have to bring the node up (compacting with DTCS), and then change it [15:27:19] i'm not sure what (if any) implications there are with this [15:27:40] right urandom ... expecting a no-restart is kinda not a good idea :) [15:28:50] joal: sorry, meetings for the next two hours [15:29:53] np mobrovac, wanted to know some more on the scap modification (mostly for me getting better at scap, ot to challenge the change :) [15:31:13] urandom: sorry forgot to mention you - last question I promise: where did you get cassandra_2.1.13_all.deb? Is it from the debian stable repo right? Just wanted to have a rollback to 2.1.12 if needed [15:32:43] (seems so from elukey@aqs1004:/usr/share/doc/cassandra$ zless changelog.gz) [15:32:46] hiii! [15:32:55] elukey: FYI am running election to bring back 1013 [15:32:56] ottomata: o/ [15:33:00] sureee [15:33:11] what was wrong yesterday with the retention ms? [15:33:21] I saw that you made it work [15:33:31] oh yeah [15:33:34] uh, i think i just had a bad value [15:33:41] some how my multiplication and division was off [15:33:46] i had set it higher! not lower :p [15:33:53] dunno why bytes didn't work [15:34:06] but with a actual 48 hour ms value, it deleted logs [15:34:41] all right, might be useful in https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Kafka/Administration#Purge_broker_logs whenever you have time [15:35:16] jaaa thanks, editing now [15:35:19] you da best :) [15:36:02] da best annoying colleague :P [15:37:19] * elukey brb! [15:39:08] Analytics-Tech-community-metrics: List of Phabricator users - https://phabricator.wikimedia.org/T37508#2309485 (Qgil) [15:40:45] Analytics-Kanban: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2309486 (Nuria) @Gwicke: are you able to provide sample kandemila config? [15:43:13] elukey: done! https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Kafka/Administration#Purge_broker_logs [15:46:13] ottomata: niceeeee thanks! [15:47:22] elukey: http://dl.bintray.com/apache/cassandra/pool/main/c/cassandra/ [15:47:58] urandom: how many beers do I owe you? :D [15:48:22] elukey: heh [15:48:26] thanks :) [15:49:33] elukey: no worries; happy to help! [15:58:45] Analytics-Kanban, Patch-For-Review: Augment oozie load SLA + Add URL to oozei error messages - https://phabricator.wikimedia.org/T134876#2309549 (Nuria) Open>Resolved [15:59:04] Analytics-Kanban: Get jenkins to automate releases {hawk} - https://phabricator.wikimedia.org/T130122#2309553 (Nuria) [15:59:06] Analytics-Kanban, Continuous-Integration-Config: Add a maven-release user to Gerrit {hawk} - https://phabricator.wikimedia.org/T132176#2309552 (Nuria) Open>Resolved [15:59:55] Analytics-Kanban, Patch-For-Review: Pageview definition bug for apps pageviews on rest endpoint - https://phabricator.wikimedia.org/T135168#2309554 (Nuria) Open>Resolved [16:00:23] Analytics, Revision-Slider, TCB-Team, WMDE-Analytics-Engineering, and 2 others: Data need: User Behaviour when comparing article revisions - https://phabricator.wikimedia.org/T134861#2309556 (Tobi_WMDE_SW) p:Triage>Normal [16:00:44] a-team: ahem.. standuppp [16:00:50] :) [16:08:55] Analytics-Kanban: Figure out if the Changelog file can be updated in the release process by Jenkins {hawk} - https://phabricator.wikimedia.org/T132181#2309579 (madhuvishy) [16:09:26] Analytics-Kanban: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2309587 (Nuria) a:Milimetric>Nuria [16:09:55] Analytics-Kanban, Patch-For-Review: Create repo analytics.wikimedia.org with index and build of browser reports for puppet to source and deploy to analytics.wikimedia.org - https://phabricator.wikimedia.org/T134506#2309593 (Nuria) a:Nuria [16:35:20] madhuvishy: coming to tasking? [16:35:58] nuria_: coming in 5 mins, finishing the upgrade [16:37:20] oh tasskibg [16:38:38] Analytics: Extract edit oriented data from MySQL from simplewiki (small size) - https://phabricator.wikimedia.org/T134790#2309700 (Nuria) [16:39:32] Analytics: Extract edit oriented data from MySQL for small wiki - https://phabricator.wikimedia.org/T134790#2277147 (Nuria) [16:55:13] Analytics: Extract edit oriented data from MySQL for small wiki - https://phabricator.wikimedia.org/T134790#2277147 (Nuria) First question: 1) Do we want to move data to event bus first or rather we want to go directly to analytics schemas? Given that our goal is to be able to have a prototype of data pip... [17:20:07] Analytics: Extract edit oriented data from MySQL for small wiki - https://phabricator.wikimedia.org/T134790#2309855 (Nuria) The DB loading is assumed to be a bootstrapping step, to happen only once. Updates to the past data that are happening to db data should come as eventbus events so we are not consideri... [17:23:13] Analytics: Extract edit oriented data from MySQL for small wiki - https://phabricator.wikimedia.org/T134790#2309857 (Nuria) [17:23:22] Analytics: Extract edit oriented data from MySQL for small wiki - https://phabricator.wikimedia.org/T134790#2277147 (Nuria) [17:25:24] Analytics: Extract edit oriented data from MySQL for small wiki - https://phabricator.wikimedia.org/T134790#2309864 (Nuria) [17:27:44] Analytics: Extract edit oriented data from MySQL for small wiki - https://phabricator.wikimedia.org/T134790#2309876 (Nuria) An idea on how to approach that task: http://www.gv.com/sprint/ [17:29:42] Analytics-Kanban: Extract edit oriented data from MySQL for small wiki - https://phabricator.wikimedia.org/T134790#2309884 (Nuria) [17:32:59] Analytics, WMDE-Analytics-Engineering: Remove http://datasets.wikimedia.org/aggregate-datasets/wikidata/ - https://phabricator.wikimedia.org/T125407#1986786 (Nuria) We need to: - see if anyone is using this data - see if there are any crons creating data - removing files entirely - removing apache access [17:35:00] Analytics-Kanban, WMDE-Analytics-Engineering: Remove http://datasets.wikimedia.org/aggregate-datasets/wikidata/ - https://phabricator.wikimedia.org/T125407#2309892 (Nuria) [17:48:26] Analytics-Kanban, Datasets-General-or-Unknown, WMDE-Analytics-Engineering: Fix permissions on dumps.wm.o access logs synced to stats1002 - https://phabricator.wikimedia.org/T134776#2276672 (Nuria) [17:49:57] Analytics: Describe threat model for sanitized pageview data {mole} - https://phabricator.wikimedia.org/T131158#2309967 (Nuria) [17:51:10] Analytics-Cluster, Operations, ops-eqiad: kafka1013 hardware crash - https://phabricator.wikimedia.org/T135557#2309973 (Ottomata) [17:56:54] Analytics: Describe threat model for sanitized pageview data {mole} - https://phabricator.wikimedia.org/T131158#2310009 (Nuria) Goal: "If we release the dataset with and without page title info. what are exploits that we know are possible (cross-checking with other datasets)?" Compute a month of data we cou... [17:58:28] Analytics-Kanban: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2310013 (GWicke) @nuria: Again, T135240#2302880 has a link to a sample config, and instructions on which values to set for the peers. If you would like us to set up & deploy a config for you, then I think we c... [17:59:24] a-team: logging off, talk with you tomorrow! I'll check later on aqs but look goot atm [17:59:27] byyyeeee [17:59:34] bye elukey ! [17:59:35] elukey, bye! [17:59:44] (tomorrow I'll migrate aqs100[23]) [18:00:01] byyye [18:13:56] joal: see my e-mail about xanalytics lazy value [18:14:11] joal: it cannot be used for bucketing (if that is what it was intended) [18:14:28] joal: but it can report the bucketing that has happened client side [18:14:39] ottomata: hi. [18:17:25] nuria_: hey you there? [18:17:36] jdlrobson: holaaa yes [18:17:43] figured might be easier to talk here :) [18:18:11] jdlrobson: sure, that works too, do read my last e-mail i just sent [18:18:15] yup [18:18:20] urandom: hiyaa [18:18:21] jdlrobson: and let me know if it makes sense [18:18:24] i figured i could clear up misunderstanding and then we could clarify [18:18:31] jaja [18:18:34] So all I want to do is be able to see there was more engagement in bucket A than bucket B (e.g. which bucket had more page views) [18:18:39] ottomata: how are you!? :) [18:18:47] bucket A would lazy load images, bucket B would not [18:18:47] jdlrobson: yes, understood [18:18:55] the hope is that through better performance we'll see more page views [18:19:12] So in my head this has nothing to do with JS [18:19:13] jdlrobson: you can REPORT bucketing with x-analaytics [18:19:20] urandom: am well! a little sore throaty today so meh, but not enough to take day off :) got my head stuck up in druid puppetization land :) [18:19:24] how are you? [18:19:27] jdlrobson: but bucketing has to be stored client side elsewhere [18:19:39] jdlrobson: how does the client know that it has to lazy load images? [18:19:40] ottomata: i'm ok [18:19:51] nuria_: the 50% a/b test will be done in varnish [18:19:53] based on IP address [18:20:05] ottomata: i'm looking for someone to exploit^H^H^H^Hto ask for a merge [18:20:13] jdlrobson: that is what i am saying you cannot dowith x-analytics [18:20:31] ottomata: you have a fetching timezone, and i think, +2 on puppet, yes? :) [18:20:47] jdlrobson: cause lazy loading of images is happening from client correct? [18:20:58] jdlrobson: or is varnish initiating teh lazy loading? [18:21:00] *the [18:21:39] nuria_: varnish. Haven't quite worked out how we propagate that down to the PHP as im waiting on the implementation from brandon black [18:21:47] hah, urandom yes indeed [18:21:51] whatchu neeeed? :) [18:21:57] https://gerrit.wikimedia.org/r/#/c/289685/ [18:22:00] but am hoping PHP will be able to read a header [18:22:14] it's for the restbase staging env, not production per say [18:22:17] jdlrobson: varnish is initating the lazy loading of images .. wait.. how? [18:22:26] ok cool [18:22:32] was about to ask, also looked it up in site.pp [18:22:33] sounds fine to me [18:22:35] ottomata: \o/ [18:22:37] can you run puppet? [18:22:40] or shall I? [18:22:41] i can [18:22:43] ok [18:22:46] jdlrobson: listening, sorry, go ahead [18:22:46] i have it disabled atm [18:23:53] nuria https://phabricator.wikimedia.org/T127883 so it was my understanding that varnish will look at the IP of the incoming user if (is_ipv4 and ipv4_string ~ '[0-4]$') { turn_on_lazy_loading } [18:24:06] ottomata: thank you sir! [18:25:14] ya puppet-merged [18:25:15] yw [18:27:04] i'm not entirely sure about the actions of turn_on_lazy_loading but am hoping it sets a header [18:27:26] nuria_: I understand your point and that idea of header ids for reporting was implicit for me [18:27:47] jdlrobson: so the php is going to look different for the users in the test? are we rendering on the php side different "" tags given a header passed from varnish? [18:27:49] nuria_: You surely can't enforce information client side based on server side event :) [18:28:24] nuria_: but as you said, you can report, and that my point [18:28:42] joal: the part i missed was that is varnish (not the client) turning on lazy loading which seems odd but not so much on our stack [18:29:42] nuria_: Would be good to double check as I suggested before turning the actual feature on that both population look the same (meaning having varnish split populations, send different headers, but no difference in code, and after av few days, activate code difference [18:29:49] jdlrobson: brandon's comment might imply that this cache bypass is only happening for logged in users though.. [18:30:25] nuria_: php html output will be different yes for 2 groups [18:30:33] the cache will be fragmented [18:30:41] for anons and logged in [18:30:49] which comment makes you think that? [18:31:13] a-team, logging off, I'll finish pageview backfiling tomorrow morning and start playing with new cassandra, but will be off in the afternoon [18:31:52] Have a good end of day and a good weekend (except for European time people that I'll see tomorrow :) [18:33:16] jdlrobson: ok, read the whole ticket and if you want a 50% split even you wouldn't get that with IPs [18:33:32] jdlrobson: but that is something else i can talk to brandon about [18:35:03] jdlrobson: let me talk to brandon about the ip strategy [18:44:33] nuria_: that would be great [18:44:54] jdlrobson: it is a great risk to launch this to 50% of our users [18:45:27] nuria_: we are ready to launch it. We are not worried about the code. We just want to measure impact at this point. You are right though it does not need to be a clear 50% split [18:46:02] the most important thing is that with regards to performance we can distinguish 2 buckets of users who use a cache populating by the same amount of users [18:46:18] jdlrobson: we can find out how well it works with a much smaller set of users, impact in a user base as large as ours can be stablish by pinging 10% of users [18:46:23] which i guess would mean we need group a, group b and control where control = 80%, A and B are 10% each [18:47:01] cos if A is using a cache populated by 90% of users and B is using one cached by 10% we have a problem measuring performance impact [18:57:03] Analytics-Kanban: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310174 (Nuria) [18:57:12] Analytics: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310186 (Nuria) [19:05:58] Analytics, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown, and 2 others: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#2310198 (Sadads) I would think standard enabling it across the board, is going to be important, both... [19:17:17] milimetric, do you know where do the metrics starting with average- in cloud9 come from? I could not find a definition anywhere... [19:17:27] looking [19:17:42] yes, hang on [19:18:38] mforns: we added the word "average" ourselves so we could understand what they meant, because it's a big cryptic just transfering the title from where they are in wikistats without the numbers that give you an idea of what they're measuring: [19:18:39] https://stats.wikimedia.org/EN/SummaryEN.htm [19:18:50] https://stats.wikimedia.org/EN/TablesWikipediaEN.htm [19:18:57] (mostly that first one) [19:19:21] so like the row that says "New Articles per Day" would be average-new-articles-per-day [19:20:24] but they're still kind of confusing to me, mforns [19:20:50] it sounds like we'd count something every day, then sum and divide by the count, and that's the "monthly" number [19:21:09] milimetric, thanks a lot, looking into it [19:24:43] Analytics, Hovercards, Reading-Web-Backlog, Reading-Web-Sprint-72-Ninety-nine-problems-but-Nirzar-aint-one: Verify X-Analytics: preview=1 in stable - https://phabricator.wikimedia.org/T133067#2310242 (dr0ptp4kt) [19:28:04] Analytics, Hovercards, Unplanned-Sprint-Work: Capture hovercards fetches as previews in analytics - https://phabricator.wikimedia.org/T129425#2310250 (dr0ptp4kt) [19:28:10] Analytics, Hovercards, Reading-Web-Backlog, Reading-Web-Sprint-72-Ninety-nine-problems-but-Nirzar-aint-one: Verify X-Analytics: preview=1 in stable - https://phabricator.wikimedia.org/T133067#2310248 (dr0ptp4kt) Open>Resolved Looks to be working based on data from the hour 2016051801 on E... [19:35:57] ottomata: so on stat1002 we have mailx available, which is super useful for auto-emailing results to yourself after a long query has completed [19:36:05] ...can we have mailx on stat1004 too? [19:39:38] milimetric: woooo finally! io.druid.indexing.worker.executor.ExecutorLifecycle: Task completed with status: { [19:39:39] \ "status" : "SUCCESS", [19:39:46] malix! [19:39:50] huh never heard of it... [19:39:51] looking [19:40:10] HaeB: do you know if it is from a .deb package? and if so, what? [19:40:24] awesome ottomata [19:40:39] so ok, milimetric, now what? [19:40:40] :) [19:40:52] oh HaeB mailx? [19:41:19] ya mailx [19:41:24] https://en.wikipedia.org/wiki/Mailx ;) [19:42:02] stat1003 has it too [19:42:18] oh HaeB sorry, i read that as malix [19:42:18] haha [19:42:25] yeah, stat1004 is a little different, but ja we can do that [19:43:21] that would be great [19:43:44] how else is it different btw? (just curious) [19:43:47] ottomata: I mean we've played with it in labs enough. I think the unknowns are mostly performance. But we do want to look at some of the new functionality like the lookups. So don't tear it down yet. [19:44:28] oh i won't tear it down, i have lots more to do, but i guess i want to verify deep storage stuff next [19:44:33] i'll turn back on mysql and hdfs storage stuff [19:44:36] try to get that all straight [19:44:46] makes sense [19:44:46] then we'll try indexing some bigger files out of hdfs [19:44:54] might need some help to test querying in a bit [19:44:56] maye tomorrow [19:45:17] yeah, we're working on exporting lots of data next I guess [19:47:05] ah rats actually, now historical is having problems [19:47:05] hm [19:47:34] memory? [19:48:38] no [19:48:38] Instantiation of [simple type, class io.druid.segment.loading.LocalLoadSpec] value failed: [/tmp/druid/localStorage/pageviews/pageviews/2015-09-01T00:00:00.000Z_2015-09-02T00:00:00.000Z/2016-05-19T19:37:17.074Z/0/index.zip] does not exist [19:48:51] HaeB: done [19:49:16] nice, thanks! [19:49:40] maybe one thing's looking in hdfs and the other's saving to disk or vice versa [19:51:10] yeah i think its a java tmp dir thing, looking into it... [19:52:46] huh weird milimetric this is because I have local storage set to noop [19:52:51] i thought it woudln't try to use deep storage then [19:52:53] clearly it is [19:53:01] the default localstorage directory is in /tmp/druid/localStorage [19:53:04] huh [19:53:33] i guess i'll make the default be local, instead of noop [19:53:36] and just set that properly [19:54:05] yeah, I never saw noop before, maybe it's a bad/old doc or something [19:56:01] ottomata: could i trouble you for one more merge? -- https://gerrit.wikimedia.org/r/#/c/289722/ [19:56:13] ottomata: same as before, but widens it from one host, to all (in rb staging) [19:56:38] urandom: this will do all of them? [19:56:41] ottomata: and baring some 'oh crap' moment where i needed to rollback, this would be the last [19:56:43] including prodones? [19:56:47] prod ones [19:56:50] just staging [19:56:56] staging in eqiad and codfw [19:57:29] ah i see _test_ ones [19:57:29] ok [19:57:47] yeah, our staging is in the prod network... but they aren't prod hosts per say [19:57:53] aye [19:58:01] done [19:58:11] ottomata: sweet; thanks again man! [19:58:14] np! [20:03:48] (CR) Jdlrobson: [C: -1] "Needs rebase" [analytics/wikistats] - https://gerrit.wikimedia.org/r/145862 (owner: Nemo bis) [20:04:16] (CR) Jdlrobson: [C: -1] "Needs rebase" [analytics/wikistats] - https://gerrit.wikimedia.org/r/118261 (owner: Nemo bis) [20:18:14] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310391 (BBlack) From irc conversation in wikimedia-operations w/ @Nuria, @Krinkle, and myself. This is the varnish-level pseudo-code proposed (ignore arbitrary names and con... [20:26:40] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310419 (Nuria) Looks great, one minor nit: rather than having distinct cookies per feature we can have one weblab cookie that contains all features and bucketing for those.... [20:26:48] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310422 (BBlack) Other nits and notes: 1. Don't send the cookie to the applayer, just the feature header 2. Validate the cookie's value, clear+reset if invalid 3. Block the f... [21:05:25] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310635 (BBlack) Better pseudo-code, after more conversation: Data structure (which we update as we add/remove experiments): ``` experiments => { # 100 total bins to use: 0-9... [21:10:21] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310677 (BBlack) If the `setRequestHeader` bits didn't use else-if, you could have overlapping buckets with multiple features in play too, but this seems simpler for the momen... [21:11:42] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310694 (BBlack) Another complication that didn't come up in conversation earlier: what about domainnames for these cookies? They'll be getting binned independently for every... [21:14:47] Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2310716 (BBlack) Another thought: with the above code, I've intentionally set it so that both sides of the experiment will initially share an empty cache split (they'll have t... [21:39:47] Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2310840 (DannyH) @eranroz @Earwig What's an estimate of how many queries Eranbot and cop... [22:11:03] (Abandoned) Madhuvishy: Revert "Test commit for jenkins release testing" [analytics/refinery/source] (release) - https://gerrit.wikimedia.org/r/279410 (owner: Madhuvishy)