[00:58:41] 10Quarry: Provide a way to hyperlink Quarry results/output - https://phabricator.wikimedia.org/T74874#3410189 (10MZMcBride) [05:06:46] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10DBA, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3410454 (10Marostegui) As Nuria said, this can be handled by the Analytics team [07:31:51] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10DBA, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3410630 (10elukey) @kaldari we can do it! If you don't mind I'd wait for Andrew to come back from v... [07:32:50] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10DBA, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3410631 (10Marostegui) I would suggest to rename them before issuing any drop. Leave them renamed w... [07:43:05] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10DBA, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3410635 (10elukey) >>! In T169781#3410631, @Marostegui wrote: > I would suggest to rename them befo... [07:59:04] yessss fixed the two alter tables for db1047 [08:00:16] dbstore1002 is 271/314 [08:00:30] so it might finish by tomorrow [08:32:55] 10Analytics-Kanban, 10Patch-For-Review: Add a job that regularly deletes druid webrequest deep-stored data - https://phabricator.wikimedia.org/T168614#3410709 (10JAllemandou) [08:33:02] 10Analytics-Kanban, 10Patch-For-Review: Add normalized_host.project_family and deprecate and remove normalized_host.project_class - https://phabricator.wikimedia.org/T168874#3410710 (10JAllemandou) [08:36:57] 10Analytics-Kanban: Troubleshoot issues with sqoop of data not working for big tables - https://phabricator.wikimedia.org/T169782#3410722 (10JAllemandou) @milimetric: Thanks for clarifying context, I confirm the sequence you rebuilt. Some more info: I tried a few manual runs against enwiki.revision --> They all... [08:38:32] Hi a-team - I just double checked the last thing I wnated to do (add context to sqoop failing task) - milimetric got it all sorted yesterday [08:38:44] Thanks a lot for handling my mess :) [08:39:01] o/ [08:39:11] I'll be off from now on as said, and will send an email once I get a good picture of my new loved one [08:39:31] enjoy your family! [08:39:40] elukey: :D [08:39:56] elukey: I managed to have 6 hours of sleep over two nights - I feel better ;) [08:41:27] :D :D [08:48:59] see you all my team soon - I miss you already ;) [08:49:09] we'll miss you too Jo! [08:49:12] have a nice time off! [09:48:10] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3410859 (10elukey) Alter tables completed on db1047, these are the remaining tables with NOT NULLABLE columns that are NOT whitelisted afaics: ``` MobileWikiAp... [10:13:59] hellooo [10:23:25] o/ [10:37:24] !log taking mysqldump for Piwik and storing it on stat1002:/a/backup/bohrium/mysqldump_20170706.sql [10:37:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:43:39] really nice thing that now bohrium is not contantly under IOWAIT [10:43:47] and the mysqldump is 10G [10:44:06] (it was more than 20G before) [10:53:31] cool! [10:57:59] mforns: did you see my last post for the nullable cols? [10:58:09] elukey, ... no [10:58:19] where? task? looking [10:58:24] https://phabricator.wikimedia.org/T167162#3410859 [10:58:58] yea reading [10:59:39] "Alter tables completed on db1047" \\\o/// [10:59:48] is this the fast one? [10:59:57] nope [10:59:58] I always ask the same question :] [11:00:02] oh! ok [11:00:51] elukey, and only those 4 tables are not "nullableified" ? [11:01:05] we can try but it takes ages [11:01:48] wait, but you sain in a prior post that there were like 17 tables that were not being nullableified [11:01:52] *said [11:03:41] mforns: I removed the ones that have cols whitelisted [11:04:22] elukey, you mean the ones that do NOT appear in the whitelist? [11:04:59] "these are the remaining tables with NOT NULLABLE columns that are NOT whitelisted" [11:05:06] aaaah! [11:05:12] :) [11:05:21] ok, so you did all the analysis already, awesome :D [11:06:53] elukey, how about the other db? [11:07:09] the other slave host [11:07:20] will redo the analytis after the alter tables are completed, probably tomorrow [11:07:55] mforns: I was wondering if the cols that I put in the task are really sensitive or not [11:08:13] elukey, they are [11:08:13] just to know if we need to do the remaining big alters or not [11:08:16] ah okok [11:08:27] there is one query that will surely not finish [11:08:34] MobileWikiAppArticleSuggestions_11448426_15423246 [11:08:41] it holds 200M+ rows [11:08:50] so db1047 will not be able to do it [11:08:54] elukey, maybe we can ask the owners if we can drop the 50% of the records (the oldest data) [11:09:07] aha [11:09:13] that would be awesome [11:11:09] elukey, ok will write [11:11:38] elukey, unless... [11:12:48] 81 GB ? O.oU [11:13:01] 1 single table? [11:13:03] OMG [11:14:25] I am attempting to alter the 40G one :D [11:14:35] ok :] [11:14:44] elukey, is it possible to mysqldump into kafka? [11:15:08] mmmm why? :D [11:15:14] hehehe [11:15:39] maybe we could mysqldump MobileWikiAppArticleSuggestions_11448426_15423246 into a kafka topic, and then drop the table [11:15:46] recreate it with the nullable columns [11:15:56] and then import all the data from kafka again [11:16:59] mmmm [11:17:36] well, it's worth try and ask reading if the data can be shortened [11:22:43] well we can use a regular disk, 80G is not that much [11:33:28] * elukey lunch! [11:37:58] (03PS1) 10Filippo Giunchedi: Add scap3 config [analytics/statsv] - 10https://gerrit.wikimedia.org/r/363578 (https://phabricator.wikimedia.org/T129139) [11:39:36] ok, definitely.... we shouldn't use pageviews.js in a client-side javascript project [11:40:18] previous uncompressed bundle size: 4.10Mb [11:40:32] --removed pageviews.js, replaced it with direct api calls--- [11:41:08] current uncompressed build size: 2.16Mb [11:41:29] (03PS2) 10Filippo Giunchedi: Add scap3 config [analytics/statsv] - 10https://gerrit.wikimedia.org/r/363578 (https://phabricator.wikimedia.org/T129139) [11:41:52] gzipped, minified: 258kb so already at your benchmark milimetric :D [11:42:26] (03CR) 10Filippo Giunchedi: "Puppet part at https://gerrit.wikimedia.org/r/#/c/363579/" [analytics/statsv] - 10https://gerrit.wikimedia.org/r/363578 (https://phabricator.wikimedia.org/T129139) (owner: 10Filippo Giunchedi) [11:48:54] fdans, the pageviews.js lib makes 2MB??? OMG [11:49:15] it's because of it's use of request [11:49:23] oh! [11:49:26] it's supposed to be used in server with node [11:49:30] not in the client [11:49:31] aha [11:49:47] it would be cool to make a fork using $.get instead of request [11:49:57] makes sense fdans :] [11:50:00] so that it's usable on the web [11:50:08] we should remove it from dashiki too :) [11:50:20] fdans, good catch [11:50:43] it was thanks to this awesome tool, webpack-bundle analyzer [11:50:50] generates this map: [11:51:01] https://usercontent.irccloud-cdn.com/file/qxFqWOCn/Screen%20Shot%202017-07-06%20at%2013.09.16.png [11:51:16] ^ this is before [11:51:40] https://usercontent.irccloud-cdn.com/file/TdI1f0Ih/Screen%20Shot%202017-07-06%20at%2013.51.24.png [11:51:43] and this is after [11:51:45] crazy [13:01:34] milimetric: I think we can't really reduce the size of semantic [13:02:14] like, that big semantic.css things actually only contains things that we need, I think [13:12:55] fdans: semantic is modular [13:12:59] you can include only the css you need [13:13:08] it takes a little grunt work, and I'm probably the best person to do that [13:13:21] milimetric: yeah but we're only including in that file ths stuff that we specified in semantic.json [13:13:41] right, but that's just selecting everything I think [13:13:52] we can limit it to only the stuff we use [13:14:04] yeah that's what I was doing [13:14:14] oh you're saying there's no more stuff to cut [13:14:19] yeah [13:14:29] I see, let's take a look, did you push? [13:14:48] I pushed the pageviews stuff [13:15:30] I've reverted what I was doing with semantic because I cut too much and it didn't appear to be saving a lot of spce [13:16:22] regardless, yeah, the current semantic.json includes a lot of components we dont use milimetric [13:16:32] however I don't think removing them will save much space [13:16:55] that doesn't sound right :) [13:17:36] well the components vary a lot in size [13:17:50] for example we use grid, which is huge in comparison with the rest [13:18:52] right [13:18:58] back in a bit, I'm going to get something for lunch (at 3pm :P) [13:19:16] k, have fun, gonna look at this a bit and then go back to sqoop [13:35:38] milimetric: yeah don't put too much thought into it, we're at 260kb gzipped, we can deploy with this [13:36:02] oh I saved about a hundred k minified, one sec, just verifying [13:37:58] 😎 [13:38:41] fdans: ok, pushed [13:38:48] build and let's see where we're at [13:40:14] with semantic, the right way to do it is to import what each component needs and bundle different components in different bundles [13:40:30] like charts/WorldMap.vue isn't needed in the main bundle [13:41:16] oh, lemme replace lodash too, that'll get rid of the last big thing that's easy [13:43:44] hm... nah, I'll save that for later, and probably easier to customize the lodash build than to replace [13:58:21] fdans: new idea, lodash is actually super modular, I can just import only what I need and bundle it up, then import that bundle instead of 'lodash' [13:58:36] oh nice [14:23:05] hm, something's still including the whole of lodash but the bundling seems to work and I don't see any references to lodash in our code [14:23:06] maybe vue? [14:28:48] no, weird, can't find it [14:28:50] oh well [14:41:45] fdans: hola! [14:41:57] wolaaa [14:42:38] fdans: the pageviews.js does not need to bundle request if used client side, we use it in dashiki and our bundle is really small so it is not the pageviews.js module but rather bundler might be too eager [14:43:24] fdans: it is 140K compressed [14:43:46] cc mforns [14:43:54] hello [14:44:05] that's still pretty big for our current size [14:44:21] we're at... [14:44:50] I see [14:44:51] fdans: but it is not due to pageviews.js, client side is weight is really small , it doesn't need any node dependencies [14:45:22] fdans: so it is just 1 file compreseed [14:46:09] yeah pageviews.js itself is tiny, I didn't know there was a client side mode that didn't use request [14:46:32] fdans: https://github.com/tomayac/pageviews.js/blob/master/pageviews.js#L24 [14:46:34] fdans: if you look at the code it forks [14:46:44] fdans: exactly [14:46:47] that's the line that the bundler must not see [14:47:02] anyway, we don't need pageviews.js, it doesn't make sense in our case at all [14:47:12] i agree milimetric [14:47:25] it was replaced with 1 line of code, so even at 9K minified it's still a bunch of unnecessary code [14:47:26] milimetric: rather, in this case the bundler needs to allow you to overrrde package.json [14:47:37] well, yeah, that problem we need to solve, agreed [14:47:45] but I still don't think we should use pageviews.js [14:47:57] milimetric: client side? [14:48:03] yeah, it doesn't make sense, it brings nothing [14:48:28] milimetric: no, it adds argument parsing and promise hanfdling [14:48:37] milimetric: we woudl need to do those regardless [14:49:15] milimetric: is npm run build working for you right now? [14:49:38] argument parsing isn't relevant here since we have full control over the arguments and verifying them would be the job of tests if anything [14:49:51] pageviews.js makes sense in cases where you're building a UI with user input [14:51:17] also, there is no promise handling, you should take a look at the commit that replaces it nuria_: https://phabricator.wikimedia.org/rWIKISTATS95ec0257d82be62568f7cc44faf264029b166968 [14:51:30] milimetric: in this case, yes , arguments are less useful [14:51:38] fdans: it works, yea, but it outputs 800k [14:51:51] I wasn't sure if that's because it wasn't gzipped or you changed something or whatever [14:52:04] * elukey is happy about not having the problem of thinking about js [14:52:06] it's erroring for me [14:52:07] there are still problems though - the bundler is including lodash in full [14:52:22] fdans: let's hang out, standup's almost here anyway [14:52:23] elukey: 99 problems, js ain't one! [14:52:27] milimetric: sure [14:52:32] omw [14:53:14] milimetric: ok, as you wish , if you feel it does not add any value go ahead and remove it. I mainly wanted to make clear that it is not including a bunch of extra dependencies . [15:26:48] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10DBA, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3411969 (10elukey) @Marostegui: The idea would be to: 1) stop eventlogging on eventlog1001 2) sto... [15:31:13] elukey: before i deploy cause i like to be triple safe , let me take a look at beta gaian [15:31:15] *again [15:31:35] nuria_, elukey and I are in da cave [15:44:23] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 5 others: Managing size of page-create and revison-create tables in storage. Agreggation? - https://phabricator.wikimedia.org/T169898#3411985 (10Nuria) [15:55:39] 10Analytics: Discussed above - https://phabricator.wikimedia.org/T169900#3412023 (10Nuria) [15:55:56] 10Analytics: ---------------- Discussed above -------------------- - https://phabricator.wikimedia.org/T169900#3412036 (10Nuria) [16:01:44] ping milimetric mforns fdans standup? [16:02:10] mforns: dropped tables again , to be super sure [16:03:03] ok [16:09:14] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3412082 (10Nuria) [16:09:33] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#2867045 (10Nuria) Not on our end that we know of. are these tables also going to be deleted from analytics store? [16:10:58] 10Analytics-Kanban: Push mediawiki history data into labs. Public history data lake - https://phabricator.wikimedia.org/T169572#3412089 (10Nuria) [16:11:30] 10Analytics-Kanban: Push mediawiki history data into labs. Public history data lake - https://phabricator.wikimedia.org/T169572#3402184 (10Nuria) p:05Triage>03Normal [16:14:22] 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, 10DBA, and 5 others: Drop tables with bad data: mediawiki_page_create_1 mediawiki_revision_create_1 - https://phabricator.wikimedia.org/T169781#3412109 (10Nuria) [16:15:36] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 5 others: Managing size of page-create and revison-create tables in storage. Agreggation? - https://phabricator.wikimedia.org/T169898#3412111 (10Nuria) p:05Normal>03High [16:15:57] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 5 others: Managing size of page-create and revison-create tables in storage. Agreggation? - https://phabricator.wikimedia.org/T169898#3411985 (10Nuria) [16:26:29] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 6 others: Managing size of page-create and revison-create tables in storage. Agreggation? - https://phabricator.wikimedia.org/T169898#3412155 (10Pchelolo) [16:34:54] 10Analytics-Kanban: Review by legal department of text on wikistats site - https://phabricator.wikimedia.org/T163229#3412172 (10Nuria) a:03fdans [16:35:28] 10Analytics-Kanban: Review by legal department of text on wikistats site - https://phabricator.wikimedia.org/T163229#3190638 (10Nuria) [16:38:41] 10Analytics: Alarm on HDFS related script failures - https://phabricator.wikimedia.org/T168390#3412185 (10Nuria) [16:39:37] 10Analytics: Alarm on HDFS related script failures - https://phabricator.wikimedia.org/T168390#3362956 (10Nuria) [16:40:10] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3412190 (10Marostegui) >>! In T153033#3412082, @Nuria wrote: > Not on our end that we know of. are these tables also going to be deleted from analytics store? They do exist on db1047 and dbstore1002 (enwiki db... [16:49:41] 10Analytics-Kanban: Troubleshoot issues with sqoop of data not working for big tables - https://phabricator.wikimedia.org/T169782#3412334 (10Milimetric) Thanks very much, Joseph, no worries, I got it. Note to self: check that puppet running the cron command is actually formatting the date properly, and not like... [16:50:11] 10Analytics-Cluster, 10Analytics-Kanban, 10User-Elukey: Understand Kafka ACLs and figure out what ACLs we want for production topics - https://phabricator.wikimedia.org/T167304#3412336 (10Nuria) [16:53:17] 10Analytics-Cluster, 10Analytics-Kanban, 10User-Elukey: Understand Kafka ACLs and figure out what ACLs we want for production topics - https://phabricator.wikimedia.org/T167304#3412386 (10Nuria) Figuring out out the configuration (access control list) for teh clients, without stablishing any policies. Mostly... [16:54:45] 10Analytics-Cluster, 10Analytics-Kanban, 10User-Elukey: Understand Kafka ACLs and figure out what ACLs we want for production topics - https://phabricator.wikimedia.org/T167304#3412403 (10Nuria) [16:58:15] * elukey afk! [16:58:18] (03CR) 10Thcipriani: [C: 031] Add scap3 config [analytics/statsv] - 10https://gerrit.wikimedia.org/r/363578 (https://phabricator.wikimedia.org/T129139) (owner: 10Filippo Giunchedi) [17:53:48] bye team, off for today! [20:00:54] milimetric: let me know if you need a ruberduck [20:01:06] * rubber [20:02:44] nuria_: thanks, um, I gotta try something first, if that doesn't work, yeah [20:02:47] but... so slow [20:03:04] milimetric: ya, it's the bigest meta-problem [20:03:15] getting feedback on what is wrong takes forever [20:19:27] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3413278 (10Nuria) @maarostegui: then , the best way I can think of to find out whether anyone is using them is to send an e-mail to analytics@ (give people a month respond noting date by which you will delete tho... [20:23:43] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#2867045 (10demon) A //month//?! I can't imagine there's //any// useful data to be gathered out of this. MoodBar was a complete and absolute failure. [20:29:38] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3413301 (10Nuria) @demon : a month is the standard time we give users to reply to usage requests, see some data that used this extension: https://meta.wikimedia.org/wiki/Research:MoodBar/First_month_of_activity... [20:32:19] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3413303 (10demon) /me shrugs [20:41:33] nuria_: RE: https://phabricator.wikimedia.org/T159392#3080405 – stat1002 is still jq 1.3 and Trusty. [20:42:10] Oh the hardware procure is resolved, but not the replacement yet. [20:42:12] Krinkle: yes, new hardware just arrived, so we will do the switchg next quarter [20:42:17] OK [20:42:31] Krinkle: replcement will be long as there are many "things" to move [20:42:39] Krinkle: it is on our goals [20:46:29] nuria_: np, I just rm'ed my local copy of jq 1.5 and then realised it was still 1.3 :-/ [20:46:32] Just put it back :) [20:46:44] Krinkle: k [20:47:05] Krinkle: what do you use 1002 for (other than looking at hive?) [20:47:10] 10Analytics-Cluster, 10Analytics-Kanban: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3413369 (10Krinkle) [20:47:13] 10Analytics, 10Operations, 10Performance-Team: Update jq to v1.4.0 or higher - https://phabricator.wikimedia.org/T159392#3413368 (10Krinkle) [20:47:27] nuria_: kafkacat eventlogging/statsv/webrequest, and hive indeed. [20:47:40] Krinkle: k [20:47:48] I forget which I do on 02 and 03. But I think both on 02 [20:48:05] Krinkle: since you are here, woudl you guys be ok with us dropping eventlogging data for navtiming older than 90 days entirely? [20:48:15] nuria_: No! [20:48:25] Krinkle: ok, that was clear [20:48:26] Just discussed it in the meeting. I'll reply to the mail [20:48:33] Krinkle: ok, sounds good [20:49:06] It's our only uncorrupted source for nav timing over long term (graphite is unreliable for that, things change too much). [20:49:15] Krinkle: i though that if the response was yes I could clear one more thing from our plate [20:49:34] e.g. for quarterly or annual review., especially to generate histograms and such, which can't do in graphite due to aggergation [20:49:58] But short answer to the mail is: dropping pageId/rev is fine after 90 days, we don't use that in long-term at all. [20:50:09] In fact, we'll probably remove it from the schema in our next version of the navtiming schema. [20:51:15] k [21:47:54] (03PS1) 10Milimetric: Implement sqooping with mappers > 1 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/363735 (https://phabricator.wikimedia.org/T169782) [22:02:33] 10Analytics-Dashiki, 10Analytics-Kanban, 10Patch-For-Review: Create dashboard for upload wizard - https://phabricator.wikimedia.org/T159233#3413678 (10Milimetric) Yeah, let's work on it when we have time. Unfortunately we have too many things on our plate so this has to get pushed back. [22:05:18] 10Analytics: Measure Community Backlog. - https://phabricator.wikimedia.org/T155497#3413684 (10Milimetric) That's a good point of view, @Whatamidoing-WMF, and generally it's correct. But if we're to be able to measure this thing at all, we need some sort of hook. Any hook that the communities deem acceptable i... [22:13:53] 10Analytics: Provide historical redirect flag in Data Lake edit data - https://phabricator.wikimedia.org/T161146#3413722 (10Milimetric) Reedy helped me find it, of course :) Thanks!!! Here it is for French, and it's in the same place in all the Messages*.php files in that same folder: https://github.com/wikime... [22:15:22] 10Quarry, 10Cloud-Services: Consider moving Quarry to be an installation of Redash - https://phabricator.wikimedia.org/T169452#3413729 (10Milimetric) Thanks @yuvipanda, looks promising. [22:19:00] 10Analytics: wikipedia.org doesn't work, www.wikipedia.org does - https://phabricator.wikimedia.org/T169513#3413733 (10Milimetric) hm, annoying, it seems to redirect fine now. It happened right when I created this task, and I tested it on a few different devices. It's also not the first time, the first time it... [22:24:07] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3413746 (10Milimetric) The thing to consider here is that the people who are likely in the loop about what's happening on the mediawiki dbs are not in the loop about research being done on those clones on dbstore... [22:31:56] 10Analytics-Kanban, 10Patch-For-Review: Troubleshoot issues with sqoop of data not working for big tables - https://phabricator.wikimedia.org/T169782#3408433 (10Milimetric) It looks like the split-by parallelization (the easy option) worked! ``` milimetric@analytics1003:~/refinery$ hdfs dfs -ls wmf/data/raw/m... [22:35:42] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3413793 (10Nuria) > I can write to the typical research lists to ask this question if you like, so this task doesn't stall. But if this situation comes up again we should think about a good process. Let's assume...