[00:25:10] (PS1) Yurik: Updated weblogs2 to create single pages [analytics/zero-sms] - https://gerrit.wikimedia.org/r/172661 [00:25:45] (CR) Yurik: [C: 2 V: 2] Updated weblogs2 to create single pages [analytics/zero-sms] - https://gerrit.wikimedia.org/r/172661 (owner: Yurik) [10:00:01] (CR) Hashar: "Whenever this change is merged, the repository will supports a flake8 tox environment. That would let us enable the Jenkins job defined by" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172194 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [14:17:42] qchris_away: hewo! [14:18:29] hewo [14:19:14] I'm on the phone right now 1sec [14:21:46] ottomata: back [14:22:32] qchris: could you make a meeting at 20:00 UTC tomorrow? [14:22:41] so late for you! [14:22:46] but i know that you are sometimes working then [14:22:46] Yes, that would work. [14:22:53] you sure, s'ok if it is too late [14:22:56] But it's ok if you meet on friday. [14:22:56] this is not an urgent meeting [14:23:00] nope, i want you there! [14:23:22] i want you to help me with this java stuff so! there! [14:23:47] Uhm. Ok :-) [14:23:56] Ja, tomorrow 20:00 UTC is fine by me. [14:24:42] ok cool [14:24:49] Thanks. [14:31:29] qchris: I am confued about /var/log/eventlogging/archive and /srv/eventlogging-logs [14:31:34] are they the same data duplicated? [14:31:39] eventlogging-logs has more [14:31:44] more history [14:33:25] I think /srv/eventlogging-logs is an artifact from this spring. [14:33:29] I have no clue about it. [14:34:02] hmm [14:34:04] I raised it with kevinator when he declared victory on the eventlogging cleanup, [14:34:10] but he did not pick the bug up :-/ [14:34:23] hmm, ok i see, so we can probably just delete that whole thing [14:34:27] yeah i see, no new files there [14:34:27] ok [14:34:56] Corresponding bug is https://bugzilla.wikimedia.org/show_bug.cgi?id=69029 [14:35:11] Darn ^ is wrong. [14:35:37] !bug 73084 [14:35:46] https://bugzilla.wikimedia.org/show_bug.cgi?id=73084 [14:35:57] ^ is the bug for vanadium's /srv/eventlogging-logs [14:36:18] wm-bot2: When did you forget about the !bug thing? [14:49:40] Analytics / Dashiki: Story: Vital Signs User selects the Daily Pageviews metrics - https://bugzilla.wikimedia.org/72740 (Kevin Leduc) [14:50:40] Analytics / Dashiki: Story: Vital Signs User selects the Daily Pageviews metrics - https://bugzilla.wikimedia.org/72740#c19 (Kevin Leduc) Per discussions last week, we changed the scope to daily page views only. Monthly page views is another story. [14:59:57] Google does not let me join the batcave :-( [15:00:10] now it worked :-) [15:01:40] ottomata: sneeeeeeaky meeting time :-D [15:12:30] yo jgage [15:12:32] yt? [15:42:21] (CR) Nuria: [C: 2] Delete cohort tags when a cohort is deleted [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/171726 (https://bugzilla.wikimedia.org/72434) (owner: Bmansurov) [15:58:11] mforns, awake now [15:58:13] will check the task [15:58:57] 180/200 days through. We MIGHT make it. [15:59:10] Ironholds, woohoo [15:59:37] so, let me know when I can help [16:03:24] shall do [16:13:40] thx [16:33:08] going to lunch [16:40:28] Analytics / Dashiki: Story: dashiki filters outliers - https://bugzilla.wikimedia.org/73316 (nuria) NEW p:Unprio s:normal a:None If data are most probably wrong we should skip that data point in the presentation. Implementation wise to filter outliers we would need a pre-processing step t... [16:52:06] qchris: do you know the name of the wharehouse database that springle has recently set up? [16:52:23] nuria__: Sorry. No. [16:52:33] qchris: ok, we'll ask [16:53:11] But wasn't this mentioned in the email? [16:53:16] I'll check. [16:54:22] nuria__: The email says �The warehouse database on analytics-store" [16:55:49] mysql:research@analytics-store.eqiad.wmnet [(none)]> show databases like '%whare'; [16:55:49] Empty set (0.26 sec) [16:55:50] That database exists and has some relevant tables in it. [16:55:58] s/whare/ware [16:56:03] s/whare/ware/ [16:56:15] also ... the % should be at the end [16:56:16] arghhh, dislixia [16:56:24] ok. [16:56:32] I will make sure i can access [16:56:38] use warehouse [16:56:41] ^ worked for me. [16:56:48] The research user has access. [16:58:18] ottomata: do you know.... if the "beta" environment has its own varnish setup? [16:59:33] i do not know [16:59:38] Hi jsahleen. I just talked to Nuria. She will get back to you to trouble-shoot the problem regarding ContentTranslation. [16:59:43] i bet it does, but i know it doesn't have pybal [16:59:45] so not sure [17:00:08] Hi leila. Thanks! Will wait to hear from her. [17:27:47] jsahleen: does beta have its own varnish instance? where are you posting your events in beta? can you send teh url? [17:27:50] *the [17:28:40] nuria__: Niklas did the beta testing. I will have to get the url from him. He is not on call today, unfortunately. [17:29:06] jsahleen: can you repro his testing? [17:30:25] nuria__: unfortunately not at the moment. he didn't leave instructions about what he did. Only the results. [17:30:40] jsahleen: can you contact another QA eng? [17:30:47] that can repro [17:31:11] nuria__: I will see if anyone is available. We are holding office hours on another channel right now. [17:32:29] jsahleen: otherwise I am not sure we will be able to help you [17:33:41] nuria__: Ok. I understand that. Unfortunately we only have Niklas part time right now. I will ping him via email and see if he can provide the information you need. [17:38:37] nuria__: Email sent. Hopefully Niklas will respond directly to your question there. [18:04:46] nuria__: Kartik, our deployment engineer, says he does not think there is any varnish cache in beta. [18:05:11] Analytics / Wikimetrics: Can not delete tagged cohorts - https://bugzilla.wikimedia.org/72434 (nuria) PATC>RESO/FIX [18:05:15] The url we are posting events to is: http://bits.beta.wmflabs.org/event.gif [18:05:16] jsahleen: then EL events will not be getting to the db [18:05:47] nuria__: Are you saying varnish is required? [18:05:49] jsahleen: so it will be worth communicating to QA that that is not a test they can run [18:06:45] jsahleen: it is through varnish that events get published [18:07:03] Ok. I imagine that is the problem then. Thanks for your help! [18:07:41] jsahleen: np, plis make sure to tell QA [18:07:49] Will do! [18:13:32] nuria__: One more question: Can you point me to more information about how EL uses varnish? Kartik would like to take a look. [18:21:25] Analytics / Wikimetrics: Windows Line endings are not handled properly - https://bugzilla.wikimedia.org/72581 (Marcel Ruiz Forns) PATC>RESO/FIX [18:23:28] jsahleen: https://git.wikimedia.org/blob/operations%2Fpuppet.git/422b2be12181a78a31de777755358324a946242e/manifests%2Frole%2Fcache.pp#L571 [18:23:34] is all the varnish magic that there is. [18:23:43] But EventLogging can do fine without varnish. [18:24:10] So I guess (I do not have access there) that beta uses EventLogging without a varnish setup. [18:24:10] Analytics / Wikimetrics: add endpoint to show invalid users as html page - https://bugzilla.wikimedia.org/60628 (Marcel Ruiz Forns) PATC>RESO/FIX [18:24:22] https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000102.eqiad.wmflabs [18:24:30] ^ looks like it could be the relevant machine. [18:25:33] qchris: Thanks for the information. Good to know before we started setting up varnish. [18:25:46] jsahleen: also, overall EL description: https://wikitech.wikimedia.org/w/images/d/d6/Eventlogging-backend.svg [18:25:58] nuria__: But that is for production. [18:26:03] nuria__ thanks [18:26:17] jsahleen: yes, that is for production [18:26:38] beta has a dedicated instance with the eventlogging role enabled. [18:27:03] So that instance should have all the information that you need. [18:27:39] (I see you have access to the beta instances, I don't. So just find the relevant database connection there, and you know where EventLogging logs to) [18:28:14] Ok. Thanks. Much appreciated. [18:30:29] qchris: but .... [18:30:36] post are going to event.gif [18:31:06] I have to do scrum of scrums now. Will check back later. [18:31:15] qchris: who is doing the work of sending those requests along? [18:31:19] jsahleen: k. Enjoy :-) [18:32:12] nuria__: they probably are ... bit in the email from jsa hleen, he wrote that Nik erabbit found the events in the beta eventlogging db. [18:33:13] * qchris rereads the email. [18:33:27] qchris: I still do not understand what is grabbing those events and sending them if not varnish [18:33:35] nuria will read e-mail too [18:34:12] Well ... anything can. The bits.beta.wmflabs.org machines will probably do this one way or the other. [18:35:22] But if the events show up it the database, it's pretty much not important whether varnish brings them to eventlogging, upd2log does, apache does or some other system does. [18:37:07] nuria__: A request 'http://bits.beta.wmflabs.org/event.gif?foo=bar' is responded by '204 No Content' [18:37:32] ya, that makes total sense [18:37:44] That looks like something is set up in beta that behaves just like aur wvarnishes [18:37:58] Regardless of whether they are varnishes or not. [18:39:58] You have access to the instance. You go in and see if you can find some /etc/eventlogging.d/consumers/mysql-* file. [18:40:19] If you do, you know the database and how to connect to it. [18:49:56] (CR) Nuria: [C: 2] "Looks good, merging. I imagine we need to merge to enable the other job referenced in the comments." [analytics/aggregator] - https://gerrit.wikimedia.org/r/172194 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [18:51:23] qchris: the instance being I-00000102.eqiad.wmflabs? [18:51:28] cause i cannot ssh there [18:52:12] You're listed as admin on https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep [18:52:28] Mhmm... What's the error message you get? [18:54:17] (Yes, that's the instance I meant) [18:54:40] i think i will be late for the showcase today [18:54:44] SoS will go late [18:54:49] Noooooooooooooooooooooooooooo! [18:55:04] hmm i could duck out maybe [18:55:04] I'll let them know. [18:55:09] Hahaha. [18:55:30] I guess we have not much to show anyways, so I am sure it is fine if you don't duck out. [18:59:16] qchris: open failed: administratively prohibited: open failed [19:04:06] nuria__: I just tried ssh and got the same error. [19:28:44] YuviPanda: Do you happen do know how to ssh into an instance of the beta cluster? [19:29:12] qchris: if you've normal proxycommand for labs setup, then just .eqiad.wmflabs should work [19:29:35] nuria__: I guess that's what gave you "open failed: administratively prohibited: open failed"? [19:29:50] you also need to be members of the deployment-prep project [19:29:58] nuria__: Has admin there. [19:30:12] right, proxycommand on .ssh/config? [19:30:22] let me find docs [19:30:30] YuviPanda: are we memebers on deployment-prep? [19:30:31] https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_.28recommended.29 [19:30:59] nuria__: you are, I could see you in https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep [19:31:42] also you probably shouldn't be using the I-* ids to ssh [19:31:45] I don't know if that ever worked [19:31:48] you need the actual instance name [19:31:53] usually deployment- [19:31:54] YuviPanda: I access lab all the time, so my proxy i think is fine [19:31:58] yeah [19:32:04] I think the I-* id might be a problem? [19:32:15] do you know the name of the machine? [19:32:32] YuviPanda: ahem... i though that was the\ name.... [19:32:52] ah, it's called the ec2id, while name is just the actual name... [19:32:57] deployment-eventlogging02.eqiad.wmnet. [19:33:02] ^ that would be the name. [19:33:10] .wmflabs, not .wmnet :) [19:33:11] but yeah [19:33:23] * qchris facepalms. Sorry. [19:34:05] YuviPanda: how.... how can you find that out? [19:34:17] YuviPanda: so i do not have to ask you this things 10 years from now [19:34:21] Look at the "DNS name" of https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000102.eqiad.wmflabs [19:34:28] yeah, I didn't find out, qchris did :D [19:34:39] also, we never really refer to them with ec2id anymore [19:34:46] YuviPanda: ahhahaha [19:34:50] nuria__: you can also look at wikitech.wikimedia.org/wiki/Special:NovaInstance [19:35:56] nuria__: I see you've logged in! [19:36:20] YuviPanda: yes! success! [19:36:25] :) [19:36:51] Thanks YuviPanda! [19:37:30] nuria__: qchris do note that due to https://bugzilla.wikimedia.org/show_bug.cgi?id=73263 puppet hasn't run in any varnish instances on betalabs for about a month, so things are slightly out of date [19:37:51] so YuviPanda then beta does have varninsh [19:38:10] indeed, we have the nginx -> varnish -> apache setup *almost* similar to prod (no LVS) [19:38:30] YuviPanda: thank you, i should have ask you 1st thing [19:38:48] cc jsahleen: beta DOES have varnish [19:39:12] nuria__: yes, I am following the conversation. So there should be no issue, correct? [19:39:22] well, I know for sure now only because I was debuggint it earlier today :) [19:39:31] things would end up on "some" beta database [19:39:32] but we need to draw up diagrams of our stuff, both in prod and beta [19:39:49] * qchris likes diagrams! [19:40:23] I see a mysql connection in this machine that must be dumping events somewhere [19:40:24] time, though... [19:41:03] nuria__: As I wrote before ... go for /etc/eventlogging.d/consumers/mysql-* [19:41:15] That should give you the connection string. [19:44:11] right qchris was just looking at that [19:44:22] Awesome. [19:47:32] mforns, eta 1 hour [19:47:45] o/ [20:19:13] YuviPanda: despite being in the admin group i do not have sudo on the machine [20:19:25] are you sure? you shoul... [20:19:27] *should... [20:19:42] what error are you getting and what command did you try? [20:19:51] https://www.irccloud.com/pastebin/o7cFLvTk [20:20:15] hmm, that's interesting [20:20:34] YuviPanda: using my password taht i use in other instances in which i have sudo [20:20:36] *that [20:20:51] yeah, it shouldn't ask you for password [20:23:47] nuria__: see -labs :) [20:37:20] YuviPanda: another question .... [20:37:57] YuviPanda: i still cannot connect to eventlogging database on that machine [20:38:05] YuviPanda: despite of having user and password [20:38:41] nuria__: in a meeting, sorry :( 1h [20:38:48] YuviPanda: k [20:40:43] hey tnegrin, guess what task literally just completed? ;p [20:40:45] mforns, ping [20:40:55] yaaah [20:41:00] oba [20:41:15] I'll have you a newly-updated MySQL table shortly! [20:41:48] ok, then I'll import it [20:42:02] * Ironholds fistbumps [20:43:57] xD [20:59:37] (CR) Mforns: [C: 1] "LGTM" (1 comment) [analytics/aggregator] - https://gerrit.wikimedia.org/r/172237 (owner: QChris) [21:01:47] (CR) Hashar: "recheck" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [21:02:50] (CR) QChris: "Yay! The flake8-bin works!" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [21:02:59] Thanks hashar! [21:03:21] Your contint setup is soooooo easy. It's amazing. Thanks! [21:03:25] qchris: that is awesome :-] [21:03:27] next [21:03:33] add moaaar test [21:03:36] and coverage :] [21:03:44] (CR) QChris: "recheck" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172196 (https://bugzilla.wikimedia.org/72740) (owner: QChris) [21:03:47] Yay! [21:03:56] the flake8-bin job will be run when you vote +2 [21:04:04] so might not be needed to recheck them all [21:04:23] Ha. I guess you're right :-) [21:04:45] * qchris got too excited. [21:04:53] qchris: openstack generate the coverage report as HTML and upload the resulting fine to some place for people to browse it. [21:05:24] http://logs.openstack.org/33/132433/3/check/python-jenkins-coverage/bc3539f/cover/ \O: [21:05:51] Mhmm. Looks interesting. [21:06:18] Will that motivate people to write more tests? I guess so :-D [21:06:44] Can we haz logs.ci.wikimedia.org? [21:25:10] mforns, 30 seconds! [21:25:23] hehe [21:25:57] mforns, database updated :D [21:25:58] woo! [21:26:13] ok! I'm on it [21:26:55] is the table the same? [21:26:57] pentahoviews? [21:28:16] Ironholds, ^ [21:28:39] mforns, yep! [21:28:44] ok, thx [21:33:27] nuria__: am back [21:33:29] still have issues? [21:33:50] YuviPanda: on interview , will get back to you , but yes cannot connect to eventlogging db [21:33:56] :D ok [21:33:59] I can give it a shot [21:35:48] nuria__: I'm going to PM you, do see after interview. [21:36:32] YuviPanda: ya, i also see taht but it seems a permits issue right? [21:36:37] us not able to see tables [21:36:53] nuria__: no, we're root, we should be seeing everything. I don't think there're tables there [21:38:13] YuviPanda: no system tables even on db? [21:38:20] ah, hmm [21:38:21] good point [21:42:29] YuviPanda: that is strange [21:42:34] it is... [22:04:00] (PS2) QChris: Add basic implementation of projectcount aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) [22:04:02] (PS2) QChris: Add switch for automatic pushing of data repo for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172198 (https://bugzilla.wikimedia.org/72740) [22:04:04] (PS2) QChris: Allow additional logging to disk for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172197 (https://bugzilla.wikimedia.org/72740) [22:04:06] (PS2) QChris: Add basic monitoring script for projectcount aggregates [analytics/aggregator] - https://gerrit.wikimedia.org/r/172196 (https://bugzilla.wikimedia.org/72740) [22:04:08] (PS2) QChris: Add total column for project aggregation CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172237 [22:04:10] (PS2) QChris: Allow to force recomputation of existing data [analytics/aggregator] - https://gerrit.wikimedia.org/r/172236 [22:04:12] (PS2) QChris: Switch to CRLF as line terminator for CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172235 [22:04:14] (PS2) QChris: Add header line to CSVs for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172234 [22:04:36] (CR) QChris: Add total column for project aggregation CSVs (1 comment) [analytics/aggregator] - https://gerrit.wikimedia.org/r/172237 (owner: QChris) [22:16:58] Analytics / Dashiki: VitalSignsUser selects Monthly Pageviews metric - https://bugzilla.wikimedia.org/73331 (Kevin Leduc) NEW p:Unprio s:enhanc a:None Instead of selecting a daily pageviews for projects, user can select the monthly pageviews per project. Same breakdown as daily apply: mobi... [22:17:52] (CR) Mforns: [C: 1] Add total column for project aggregation CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172237 (owner: QChris) [22:26:25] Analytics / Dashiki: VitalSignsUser selects Monthly Pageviews metric - https://bugzilla.wikimedia.org/73331#c1 (Kevin Leduc) Collaborative tasking on etherpad: http://etherpad.wikimedia.org/p/analytics-73331 [22:30:51] hm, qchris, i was suggesting to inherit i think [22:30:53] not just include [22:31:09] your way will work fine, but inherit might be nice, because then your variables are automatically local [22:31:46] ottomata: Sorry. My bad. You're right. [22:34:36] Btw ... not sure about whether or not to merge this. [22:34:45] The scripts for the cron job have not been merged. [22:38:39] aye, we should wait then [22:40:45] But having the review ready would be great. How does the current patch set look for you? [22:50:32] ottomata: You sent a +1 to one of the Adventures in Clusterland emails ... was that only for that one, or for both? [22:51:03] there were two!? [22:51:17] Yup. One: [22:51:22] Adventures in Clusterland 2014-10-27--2014-11-02 (Draft) [22:51:25] and one: [22:51:29] Adventures in Clusterland 2014-11-03--2014-11-09 (Draft) [22:51:41] OO i see it! [22:51:43] Shall I resend tho one without +1? [22:51:46] Ah. Ok. [22:52:17] ees good! [22:52:35] k. Then I'll send them out. Thanks! [22:53:24] hm, qchris, is there a reason to have ::aggregator and ::Monitor as two classes? [22:53:28] why not just one with [22:53:34] ::projectcounts [22:53:58] We can throw them together. [22:54:15] yeah, unless you think they will be included on separate nodes [22:54:35] hm, and [22:54:38] I guess I can merge everything in a single class then? [22:54:50] naw, the aggregator as a separate class is good [22:54:56] because you might use aggregator as a base class for another thing too [22:55:00] if you have to do some other rollup [22:55:04] (CR) Mforns: [C: 1] "LGTM" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172236 (owner: QChris) [22:55:16] that's just DRY [22:55:39] But currently ... it's used only once ... so the R in DRY is missing :-) [22:55:59] currently! but it could be used more than once, you seem to have coded the python aggregator code to be generic [22:56:08] so your aggregator puppet class is nice generic too [22:56:24] then your ::projectcounts subclass is a specific usage of aggregator [22:56:27] You're thinking BIG! :-D Ok. Let's keep it then. [22:56:56] k, hm, as for the role class [22:57:06] i like what you are thinking, but i'm a little worried about it [22:57:16] the statistics role classes are not cleanly delineated, hm [22:57:20] lemme look closer... [22:57:56] HMMMM [22:58:06] It's just thath it did not fit anything else. [22:58:15] yeah, its pretty dumb [22:58:16] It's neither private, nor www.... [22:58:25] there are only 3 roles because there are only 3 stat servers [22:58:27] But I'll gladly just add it somewhere. [22:58:33] the roles there are mapped 1:1 to the server [22:58:36] not ideal. [22:59:36] hm, i guess i'm ok with this, because the proejctcounts class has a dependency on the dataset it needs (hdfs mount) [23:00:15] and possibly, because that mount could exist elsewhere, ::aggreagator role in theory could be incldued on another node. [23:00:17] SURE. [23:00:17] ok [23:00:37] AHHH [23:00:39] i dunno though [23:00:42] :-) [23:00:47] meh. if you make this just a single ::projectcounts class [23:00:54] that's just a single class include in teh ::private role. [23:01:13] hm, i def don't think this needs its own system::role though [23:01:13] Ok, I'll add it there and document that it's not private? [23:01:28] sure. [23:01:32] k. [23:02:01] i woudl love to clean up statistics.pp classes, but i don't think it is really worth it :/ [23:02:56] By the way Yu vi reacted yesterday, I guess Yu viPanda would certainly love to see a cleanup :-) [23:03:09] But it's ok for me. [23:15:20] it would be a huge pain, it would mean so many new modules [23:15:26] and so many of them would be shortlived [23:15:50] :-D [23:15:54] some stuff on stat servers, even if puppetized, seems to only be needed for short periods of time (< 1 year?) sometimes [23:16:30] I am fine with keeping the status quo. [23:16:37] heh [23:16:41] we should clean it up... [23:16:56] but so many things to do... [23:16:58] so little time [23:17:04] yeah, that's what I mean [23:17:08] just not worth it :/ [23:17:11] many more important things [23:17:27] so we should clean it up, probably not just right now, tho [23:17:33] but new stuff should go into a module, I think [23:19:29] i'm not 100% sure about that, i think it depends on the stuff, the aggregator thing qchris just did COULD be in a module...but it is such a small specific thing, git::clone + 2 cron job, that it is hard to justify it [23:20:18] tch tch, that's how things grow big :) [23:21:29] YuviPanda: you might be right, but i've resigned the statistics classes to how they are for now. I try to keep them at least kinda sane :/ [23:21:41] :( ok