[09:13:11] (PS1) Addshore: Only send properties to graphtie that are P\d+ [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257278 [09:13:20] (PS2) Addshore: Only send properties to graphtie that are P\d+ [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257278 [09:13:30] (CR) Addshore: [C: 2 V: 2] Only send properties to graphtie that are P\d+ [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257278 (owner: Addshore) [09:23:12] (PS1) Addshore: Only send api actions that match regex wb\w+ to graphite [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257279 [09:23:22] (CR) Addshore: [C: 2 V: 2] Only send api actions that match regex wb\w+ to graphite [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257279 (owner: Addshore) [10:10:36] Analytics-Kanban: Research and validate assumptions for pageview sanitization with Research Team [8 pts] {hawk} - https://phabricator.wikimedia.org/T120640#1857777 (JAllemandou) NEW a:JAllemandou [10:52:44] (PS1) Addshore: Add script for rolling 30 change count [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257293 (https://phabricator.wikimedia.org/T119975) [10:54:15] (PS2) Addshore: Add script for rolling 30 change count [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257293 (https://phabricator.wikimedia.org/T119975) [10:56:14] (CR) Addshore: [C: 2 V: 2] Add script for rolling 30 change count [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257293 (https://phabricator.wikimedia.org/T119975) (owner: Addshore) [11:03:02] Analytics-Tech-community-metrics, Developer-Relations, DevRel-December-2015: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#1857922 (Nemo_bis) > Wondering if there is anything left to do in this task? I don't s... [14:48:07] morning! [14:48:08] nuria: yt? [14:48:13] probably not, it is early! [14:52:44] Analytics-Backlog, operations, HTTPS: EventLogging sees too few distinct client IPs - https://phabricator.wikimedia.org/T119144#1858280 (fgiunchedi) p:Triage>Normal [15:06:29] (PS1) Addshore: Add rolling_rc to daily_site_stats.sh [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257332 [15:06:37] (CR) Addshore: [C: 2] Add rolling_rc to daily_site_stats.sh [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257332 (owner: Addshore) [15:09:02] (CR) Addshore: [V: 2] Add rolling_rc to daily_site_stats.sh [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/257332 (owner: Addshore) [15:50:58] ottomata: hola! [15:51:27] hello! [15:51:34] what was I going to ask youUUuU [15:51:34] hm [15:51:44] oh, that one function mae [15:51:45] name [15:52:00] i changed retrieve_schema() to fetch_schema, which I think is a clearer name [15:52:10] i think its ready for merge if you do! [15:54:11] ottomata: k [16:45:15] Analytics-Kanban: Write hive code doing pageview data anonimisation with two tables [13 pts] {hawk} - https://phabricator.wikimedia.org/T118838#1858609 (Nuria) After meeting with team: we are going to have our anonymization strategy peer-reviewed by research before we roll out implementation. [16:52:20] Analytics-Backlog, Design Research Backlog: Bot to call global metrics to event page {kudu} - https://phabricator.wikimedia.org/T120330#1858647 (Capt_Swing) @Abit could you describe briefly why this functionality is necessary? Who will be using it, what they do now, how this solution serves their needs bet... [17:03:47] Analytics-Backlog, Analytics-EventLogging, Blocked-on-Operations: Drop tables MobileWebClickTracking_* from eventlogging db - https://phabricator.wikimedia.org/T120674#1858723 (Nuria) NEW a:jcrespo [17:12:49] Analytics-Cluster, Traffic, operations: Secure inter-datacenter web request log (Kafka) traffic - https://phabricator.wikimedia.org/T92602#1858788 (BBlack) Due to scheduling constraints and the new-ness of the TLS-capable release of kafka, we're going to rely on IPSec for this for now and revisit TLS... [17:29:15] joal [17:29:18] https://www.irccloud.com/pastebin/eRYwcqDy/ [17:43:57] nuria: you dropped off, you ok? [17:55:21] Analytics: Pageview API demo doesn't list be-tarask - https://phabricator.wikimedia.org/T119291#1858960 (Milimetric) p:Triage>High [17:55:30] Analytics: Pageview API demo doesn't list be-tarask - https://phabricator.wikimedia.org/T119291#1822809 (Milimetric) p:High>Low [17:57:45] Analytics: Check incoming requests to secure.wm.o - https://phabricator.wikimedia.org/T119274#1858980 (Milimetric) We (team Analytics) are happy to help someone learn how to query for this data in Hive. https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/Queries [17:59:30] Analytics, CirrusSearch, Discovery: Deploy refinery-camus 0.0.23 to fix partition issues with mediawiki.CirrusSearchRequestSet - https://phabricator.wikimedia.org/T118562#1859007 (Milimetric) Open>Invalid a:Milimetric This is a duplicate that was resolved in other tasks. [18:00:29] Analytics, Traffic, operations, Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#1859016 (Milimetric) [18:00:31] Analytics-Cluster, Analytics-Kanban, Easy: Update client IP in webrequest table to use IP [5 pts] {hawk} - https://phabricator.wikimedia.org/T116772#1859017 (Milimetric) [18:02:55] Analytics-Backlog: Track overall traffic, without any filtering, broken down into major categories, for internal use. - https://phabricator.wikimedia.org/T117236#1859037 (Milimetric) [18:13:37] Analytics: stats.grok.se doesn't offer mediawiki.org page view stats - https://phabricator.wikimedia.org/T111662#1859092 (Milimetric) Assigning this to Henrik for now. [18:13:53] Analytics: stats.grok.se doesn't offer mediawiki.org page view stats - https://phabricator.wikimedia.org/T111662#1859094 (Milimetric) a:Henrik [18:18:26] Analytics, MediaWiki-API, Reading-Infrastructure-Team: Load API request count and latency data from Hadoop to a dashboard (limn?) - https://phabricator.wikimedia.org/T108414#1859119 (Milimetric) Let us know when you figure out the metric / get it measured and we can help you make a dashboard. [18:20:10] Analytics, MobileFrontend: Make MobileWebUIClickTracking schema usable (too big) - https://phabricator.wikimedia.org/T108723#1859126 (JKatzWMF) @jdlrobson I would recommend removing the search events. We could also do decreased sampling (smaller %), but only if we could sample at different rates for bet... [18:20:36] Analytics, MobileFrontend, Reading Web Planning: Make MobileWebUIClickTracking schema usable (too big) - https://phabricator.wikimedia.org/T108723#1859127 (JKatzWMF) [18:21:52] Analytics: Generalize useful pageview tools - https://phabricator.wikimedia.org/T107831#1859134 (Milimetric) Now that the Pageview API is launched, we'll assign this to the Reading team. Let us know if you need support. https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageview_API [18:22:58] Analytics-Backlog, Research consulting, Research-and-Data: Too few page views for June/July 2015 - https://phabricator.wikimedia.org/T106034#1859142 (Milimetric) [18:25:03] Analytics: Number of page requests per second for enwiki - https://phabricator.wikimedia.org/T105917#1859144 (Milimetric) Open>Resolved a:Milimetric Here are the number of pageviews per day: https://vital-signs.wmflabs.org/#projects=enwiki/metrics=Pageviews (the pageview API can get more data if you... [18:29:04] Analytics, operations: analytics1013 crashed, investigate... - https://phabricator.wikimedia.org/T97380#1859172 (Milimetric) Can this be closed? Seems out of date. [18:29:50] Analytics, Traffic, operations: Fix annoying varnishncsa+initsystem issues on jessie - https://phabricator.wikimedia.org/T97351#1859176 (Milimetric) Is this still valid? [18:30:41] hey @robla [18:31:26] robla: we talked yesterday. I'm JQ, this is my IRC handle [18:33:08] you invited me to IRC stalk you but I think freenode's whois doesn't give out your full channel info anymore [18:33:18] not until I'm on those channels anyway. [18:37:47] homunq: howdy! [18:39:00] Analytics: Junk in wmf.webrequest.uri_host field - https://phabricator.wikimedia.org/T95836#1859228 (Milimetric) @Yurik, are the new normalized fields taking care of this? Can we resolve this task? [18:40:31] homunq: did you get a chance to watch the video from last week? [18:41:37] Analytics: As an end user, i'd like an up to date set of documentation on how to access our event logging data, so that I can make data informed decisions - https://phabricator.wikimedia.org/T95027#1859241 (Milimetric) Open>Resolved a:Milimetric @Tfinc, the documentation for accessing is at: https:/... [18:44:09] Analytics-Backlog, Analytics-EventLogging, DBA: Drop tables MobileWebClickTracking_* from eventlogging db - https://phabricator.wikimedia.org/T120674#1859276 (Nuria) [18:44:24] Analytics-Backlog: Dashboard Directory Design Feedback - https://phabricator.wikimedia.org/T92502#1859278 (Milimetric) [18:45:24] Analytics: Make anonymized clickstream data available to the public - https://phabricator.wikimedia.org/T91495#1859282 (Milimetric) I think this is done to some extent with the Pageview API, and we're looking on improving. But this task is a bit generic for Analytics infrastructure, so please re-tag us if i... [18:45:50] Analytics-Backlog: Publish aggregate geodumps of article pageviews - https://phabricator.wikimedia.org/T91331#1859284 (Milimetric) [18:46:16] Analytics: What is the bounce rate of wikipedia.org as compaired to xx.wikipedia.org - https://phabricator.wikimedia.org/T90743#1859286 (Milimetric) Open>declined a:Milimetric Jared's no longer here [18:47:02] Analytics-Backlog, Fundraising-Analysis: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#1859299 (Milimetric) [18:48:09] https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/Queries [18:48:21] og4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender. [18:48:22] Logging initialized using configuration in file:/etc/hive/conf.analytics-hadoop/hive-log4j.properties [18:48:22] WARNING: Hive CLI is deprecated and migration to Beeline is recommended. [18:48:22] lol [18:50:20] Reedy: slowly progressing to the `beeline` cli application [18:51:33] but for now, the `hive` cli app also works fine [18:57:35] if anyone pinged me, I have no idea, because I have like 50 Phabricator pings :) [18:57:50] just re-ping [18:58:44] Reedy: yeah, Hive CLI is totally fine, that deprecation message is a little annoying. But we are looking into Beeline and making that available. [18:59:22] Analytics-Backlog, Analytics-EventLogging, DBA: Drop tables MobileWebClickTracking_* from eventlogging db - https://phabricator.wikimedia.org/T120674#1859365 (jcrespo) [19:02:20] Analytics-Backlog, Analytics-EventLogging, DBA: Drop tables MobileWebClickTracking_* from eventlogging db - https://phabricator.wikimedia.org/T120674#1859370 (jcrespo) @Nuria, to double confirm- these are ALL TABLES that start `MobileWebClickTracking_` and all current users are ok for them to disappea... [19:02:49] Analytics-Backlog, Analytics-EventLogging, DBA: Drop tables MobileWebClickTracking_* from eventlogging db - https://phabricator.wikimedia.org/T120674#1859372 (jcrespo) [19:02:50] Analytics-Backlog, DBA: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1859371 (jcrespo) [19:03:31] Analytics-Backlog, Analytics-EventLogging, DBA: Drop tables MobileWebClickTracking_* from eventlogging db - https://phabricator.wikimedia.org/T120674#1859373 (jcrespo) p:Triage>High [19:04:05] Analytics-Backlog, Analytics-EventLogging, DBA: Drop tables MobileWebClickTracking_* from eventlogging db - https://phabricator.wikimedia.org/T120674#1858723 (jcrespo) Usually dropping tables is considered low priority, but setting it to high as it may help with T120187. [19:24:08] robla: I watched about half of it so far; interesting. [19:24:21] you around? [19:29:53] Analytics-Backlog, DBA: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1859478 (Nuria) @Jcrespo, ori: I can help plan this work as needed. Let me know when it is a good time for ops and we will announce a partial... [19:34:38] madhuvishy: yt? [19:35:23] nuria: getting to office, will be around in 15 minutes [19:36:53] sounds good [19:38:46] Reedy milimetric ebernhardson : BTW, as far as I know we still haven't figured out how to raise the heap size limit when using Beeline, analogously to https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/Queries#Out_of_Memory_Errors_on_Client for Hive... [19:39:08] ...that means that Beeline isn't usable yet for many larger queries (that memory error isn't rare) [19:39:44] yep, true [19:39:53] Analytics-Backlog, DBA: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1859527 (jcrespo) Here is what I can give you as a feedback. Things I need: * Upgrade db1046 to the latest MySQL version (that means stopping... [19:49:28] PROBLEM - Packetloss_Average on analytics1026 is CRITICAL: packet_loss_average CRITICAL: 14.1560366667 [19:50:09] nuria: I'm around [19:50:32] madhuvishy: holaa, so how do you access your form? [19:50:44] madhuvishy: holaa, so how do you access your *new* form in wikimetrics? [19:50:48] nuria: ah yes [19:50:59] http://localhost:5000/reports/program-metrics/create/ [19:51:46] is it going to have an option on main menu or will we access it this way? [19:51:47] nuria: i've also added dan's new metric so will push latest patch in a few minutes [19:51:59] nuria: hmmm, i haven't talked about that yet - not sure [19:52:28] milimetric: what do you think? it seems like they don't want the form permanently - so not sure [19:52:29] madhuvishy: does that url go with thsi changeset: https://gerrit.wikimedia.org/r/#/c/254887/5 [19:52:39] madhuvishy: or the one after? [19:53:00] nuria: the one after - that one doesn't have the controller [19:53:22] nuria: you only need this one - https://gerrit.wikimedia.org/r/#/c/253750/ [19:53:36] * milimetric looking [19:55:25] madhuvishy / nuria: good question, I kinda don't want to add a permanent link to it somewhere, because it would confuse other users [19:55:40] ya i have the same concern [19:56:45] yeah i would just leave it as a link that someone would have to know to get to [19:56:51] we can then add it in the documentation [19:57:20] milimetric: okay cool, i'll put it in the commit message too [19:57:25] k [19:57:43] I'm gonna delete some records we don't need in the report table, I think that table's getting crazy slow because of the recurrent reports again [19:58:20] hm... wikimetrics1.eqiad.wmflabs is taking forever to ssh to [19:59:40] (PS15) Madhuvishy: [WIP] Setup celery task workflow to handle running reports for the ProgramMetrics API [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/253750 (https://phabricator.wikimedia.org/T118308) [19:59:55] nuria: pushed latest on top of Dan's patch [20:00:19] Analytics-Backlog, DBA: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1859651 (Nuria) I see, can we do this in stages? Stage 1: -Upgrade db1046 to latests Mysql. We can do this as early as Wednesday, we just ne... [20:00:30] Analytics-Backlog, DBA: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1859652 (Nuria) I see, can we do this in stages? Stage 1: -Upgrade db1046 to latests Mysql. We can do this as early as Wednesday, we just ne... [20:01:09] madhuvishy: Ok, so to see latest i have to get this one then: https://gerrit.wikimedia.org/r/253750 [20:01:17] yes [20:01:29] RECOVERY - Packetloss_Average on analytics1026 is OK: packet_loss_average OKAY: 3.33353151515 [20:04:08] nuria: also drop database wiki_testing if tests fail on something like archive_userindex.ar_page_id not found [20:05:20] Analytics-Backlog, DBA: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1859659 (jcrespo) Seems ok to me, although as I mentioned, I would prefer to schedule Stage 1 for next week. Stage 2 and the rest should be b... [20:15:13] robla: ping [20:21:38] nuria: , yt? [20:21:58] and/or madhuvishy maybe? [20:22:03] need brain bounce re python logging stuff [20:23:42] Analytics-Backlog: delete useless wikimetrics.report records - https://phabricator.wikimedia.org/T120713#1859767 (Milimetric) [20:25:33] OH, nm, no need for brain bounce [20:25:34] i am a dummy [20:34:11] Quarry, Labs, Labs-Infrastructure, HTTPS: Quarry should be HTTPS-only - https://phabricator.wikimedia.org/T107627#1859818 (chasemp) p:Triage>Normal [20:36:03] oh, hi homunq [20:37:17] homunq: I'm prepping for a couple hours of discussion I'll be having starting in 23 minutes. what's up? [20:39:37] Analytics-Backlog, Design Research Backlog: Bot to call global metrics to event page {kudu} - https://phabricator.wikimedia.org/T120330#1859835 (Abit) The business case is here: https://docs.google.com/presentation/d/1A-0rBox8tl8pL-oqab-E1Mxa2xJztHOxUWu1sFUmxPY/edit#slide=id.g2e10e3781_1159 The bot should... [20:42:44] a-team: sorry, my wifi dropped for like 30 mins [20:42:48] ottomata: here [20:43:04] ottomata: but looks like you do not need me no more [20:43:13] ottomata: sorry was eating. he he ^ [20:47:26] Analytics-Backlog, DBA: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1859851 (Nuria) >Seems ok to me, although as I mentioned, I would prefer to schedule Stage 1 for next week. Ok, let's schedule stage 1 Tuesday... [20:48:06] robla: when's a good time? [20:48:08] ottomata: can you double check what i was just telling to jaime in this ticket: https://phabricator.wikimedia.org/T120187#1859851 [20:48:12] ottomata: regarding updates top EL db [20:49:31] hi homunq: it all depends on which direction you'd like the conversation to go. let's chat in private message [20:51:59] nuria: which part do you want me to confirm? [20:52:46] Analytics-Backlog, DBA: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1859866 (jcrespo) Tuesday is ok. We should **really** avoid the application from creating more InnoDB tables. If not, this conversion will be... [20:56:00] Analytics-Backlog, Design Research Backlog: Bot to call global metrics to event page {kudu} - https://phabricator.wikimedia.org/T120330#1859877 (Nuria) @Abit: There are several tasks on that page, several of them. One is clear and it is that wikimetrics should make easier for users to calculate metrics th... [20:58:56] Analytics-Backlog, DBA: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1859882 (Nuria) >We should really avoid the application from creating more InnoDB tables. If not, this conversion will be needed again, and ag... [20:59:16] ottomata: does the overall stage1/stage2 plan sound good? [20:59:43] ja stage 1 sounds like we can just stop the mysql consumers on eventlog1001, right? [20:59:51] hopefully they will just backfill as soon as we start them bak up [20:59:52] ja? [21:00:32] stage 2 sounds like it can be done anytime after that, but it sounds like jaime wants us to fix eventlogging db code to create tokudb tables when a new schema event comes in [21:00:33] asap [21:00:33] ja? [21:01:30] ottomata: but can we even do that? create a toku db? i think that is transparent to us once teh db has been switched to the toku db engine right? [21:03:06] Analytics-Backlog, DBA: Pending maintenance on the eventlogging databases (db1046, db1047, dbstore1002, other dbstores) - https://phabricator.wikimedia.org/T120187#1859885 (Nuria) >We should really avoid the application from creating more InnoDB tables. If not, this conversion will be needed again, and ag... [21:03:22] ottomata: letme see... [21:04:27] nuria: am not sure [21:04:32] mforns might know more there [21:04:47] eventlogging does create the tables [21:04:57] ottomata: using sqlalachemy though [21:05:02] ja nuria [21:05:02] ottomata: and teh mysql driver [21:05:04] # A dictionary mapping database engine names to table defaults. [21:05:04] ENGINE_TABLE_OPTIONS = { [21:05:04] 'mysql': { [21:05:04] 'mysql_charset': 'utf8', [21:05:04] 'mysql_engine': 'InnoDB' [21:05:04] } [21:05:04] } [21:05:05] *the [21:05:06] in jrm.py [21:06:12] so....is toku db available already to be used, let me ask jaime to join [21:06:30] i think that's why we need the mysql upgrade [21:06:37] so ja stage 1 we can go forward with whenever [21:06:43] but not sure [21:07:09] nuria: since you are doing goals stuff (right?) [21:07:24] i am thikning we should make all of eventlogging use jessie + systemd next quarter [21:07:27] can we fit that in there? [21:08:33] ottomata: yes, i publish a draft here: https://www.mediawiki.org/wiki/Wikimedia_Engineering/2015-16_Q3_Goals#Analytics [21:09:13] ja would put it under the focus one [21:09:14] ja? [21:09:15] ottomata: that fits under ops excellence but it is too small to be a goal I think [21:09:22] ya [21:09:25] ok sounds good [21:09:36] ottomata: it is part of keeping lights up on ops [21:10:21] k [21:10:22] cool [21:10:56] OO you merged el patch, cool [21:10:57] ! [21:10:57] danke [21:10:59] ottomata: added it as an example of operational excellence [21:11:30] what say you about merging into service branch, then redeploying beta from master, then deploying to eventlog1001 tomorrow? [21:11:44] can I merge into master and deploy it to beta now? [21:12:47] hi [21:13:33] hiiii [21:13:44] Analytics-EventLogging, MediaWiki-extensions-WikimediaEvents, Performance: Convert WikimediaEvents statsv.js to use sendBeacon - https://phabricator.wikimedia.org/T112843#1859925 (Krinkle) [21:14:08] jynus: see my latest comment on ticket [21:14:19] jynus: for eventlogging our engine config is real meager [21:14:38] jynus: but we should try the innodb->toku db changes on beta labs before right? [21:15:11] I do not think that will work on beta [21:15:27] or CI- they have worse configuration there [21:15:54] jynus: so any ideas as to how can we test thsi before we roll it out? [21:15:57] *this [21:15:58] that should be a 1 line change [21:16:00] cc ottomata [21:16:04] either it works or now [21:16:11] I can help [21:16:29] the question is that I cannot set the default engine as Toku [21:16:45] on the server because in other servers, eventlogging is not the only content [21:16:49] jynus: but if it doesn't ... is there a way to rollback to innoDb, are both going to exists? [21:17:00] rollback? [21:17:19] rollback the config to create innodb tables agaian [21:17:21] *again [21:17:30] (CR) Madhuvishy: "+1. Just some minor comments." (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/174773 (https://bugzilla.wikimedia.org/73072) (owner: Mforns) [21:17:31] you know that most of the largest tables are already in tokudb, right? [21:17:40] jynus: no, i did not [21:17:46] it is on the ticket [21:18:10] https://phabricator.wikimedia.org/T120187 [21:18:21] ENGINE column [21:18:38] jynus: ya, i see it now [21:18:42] the question is that those were converted afterwards [21:19:01] we have to guarantee that those are created in the first place as tokudb [21:19:08] so we do not need the conversion again [21:19:12] jynus: so.. do you want to 1) update to latest mysql 2) try to make eventlogging create tokudb tables? [21:19:25] jynus: i am really not that fond of testing this in production [21:19:45] well, you can test on beta [21:19:52] but you will need to fix beta first [21:20:01] talk to RelEng [21:20:39] (which will do nothing and just tell me to do it, so not sure if useful :-) ) [21:20:42] jynus: Can you write a ticket to explain what needs fixing? I am not sure whether is the mysql engine or something else [21:20:55] jynus: ay a... [21:20:58] they need to upgrade mysql and use mariadb [21:21:07] nuria: don't we have our own mysql instance for El on beta? [21:21:19] checking.. [21:21:31] yes, but they use their own packages, not the ones on production [21:21:37] oh [21:21:40] well, i mean [21:21:45] we could change it for this instance [21:21:52] ottomata: the cli runs on our host but i am not sure where the real db is [21:21:52] since it is a single instance just for EL [21:21:56] its on our host [21:22:12] question is, db1046 is on a timer [21:22:16] ja is Version: 5.5.46-0ubuntu0.14.04.2 [21:22:59] https://grafana.wikimedia.org/dashboard/db/server-board?panelId=17&fullscreen&from=1440501127031&to=1448280487032&var-server=db1046&var-network=eth0 [21:23:26] this one is better: [21:23:28] https://grafana.wikimedia.org/dashboard/db/server-board?from=1446931397231&to=1449523157231&var-server=db1046&var-network=eth0 [21:23:38] arg [21:24:09] this one: https://grafana.wikimedia.org/dashboard/db/server-board?from=1446931397231&to=1449523157231&var-server=db1046&var-network=eth0&panelId=17&fullscreen [21:24:23] jynus: and having everything in tokudb will help with that? [21:24:36] yes, no fragmentation and 10x compression [21:24:51] the main issue here is the Edit table, which was created as new in InnoFB [21:25:07] and takes most of the space, even if it is not the largest table [21:25:12] jynus: can we switch mysql then on this host: deployment-eventlogging03.eqiad.wmflabs and try toke creation? [21:25:15] k, so EL should stop creating InnoDB [21:25:25] is there anything special about creating a table that way other than specifying engine? [21:25:26] yep [21:25:36] ottomata: right, Understood but we need to try it right? [21:25:38] yes, there are a couple of options [21:26:06] hopefully those are easy to set as is with sqlalchemy... :) [21:26:13] ENGINE=TokuDB + the compression, will give you an exact creation option line when I check which is the optimal option [21:26:44] ottomata: that is why we need to try [21:27:21] but it is very easy to test- you do not create new tables everyday, don't you? [21:27:32] just create a fake one and test it [21:27:38] jynus: with the version that is there? [21:27:42] or do we need a new version? [21:27:46] on production? [21:27:49] no beta [21:27:59] currently [21:28:00] Version: 5.5.46-0ubuntu0.14.04.2 [21:28:10] mysql-server [21:28:25] that will not work, mysql is not compatible with tokudb [21:28:27] good news...sorta...this mysql db is not puppetized in beta [21:28:35] so we can just stop it, uninstall it, and install whatever [21:28:43] are the data files compatible with whatever? [21:28:47] what version do we need? [21:29:02] any mariadb version will work, I would recommend 10 to match production [21:29:06] k [21:29:22] wmf-mariadb10 ? [21:29:34] that is what we use in production [21:29:34] (PS8) Madhuvishy: [WIP] Daily and monthly uniques oozie jobs based on WMF-Last-Access cookie, and the nocookie header in X-Analytics. [analytics/refinery] - https://gerrit.wikimedia.org/r/216341 (https://phabricator.wikimedia.org/T92977) [21:29:44] k its avail, and it should work if I uninstall mysql, and install that, right? [21:29:47] over the same /var/lib/mysql files? [21:29:56] yes [21:30:10] ottomata: ok, let me look at the toku config for mysql alchemy [21:30:14] k, nuria i'm going to stop mysql, make a backup copy of the datadir, uninstall mysql, install mariadb [21:30:17] and then start it [21:30:20] no need to rush, we can schedule this [21:30:25] on beta, this is no prob [21:30:26] can just do [21:30:28] ottomata: ok, we can drop a mobile table and recreate it [21:30:34] when I said db1046 is on a timer [21:30:38] I meant weeks [21:30:41] not hours :-) [21:30:46] hehe, i'm in here now, so it won't hurt :) [21:30:47] jynus: no, the idea is that when we do it in prod we KNOW it works [21:30:52] i just finished up a nother thing i was working on [21:30:55] so you have my attention :) [21:31:08] let me look mysql config thing [21:31:15] !log temporarily stopping eventlogging and mysql in beta [21:33:03] nuria: tell me also if you are cool with me merging the EL service branch into master [21:34:05] ottomata: ya, it's the one deployed on beta right? [21:34:09] ja [21:34:12] https://www.irccloud.com/pastebin/wl8PibgW/ [21:34:25] k, let's test this tokedb thing now taht we are on a roll [21:35:11] madhuvishy: with your patch several tests fail that are not failing on master, i can help you look at those once we do a bit of research on the toku db thing [21:35:39] nuria: ah okay, i din't run the whole thing - will check in a bit. np [21:36:03] wait, I am checking the compression [21:36:16] the documentation is contradicting itself [21:36:24] Analytics: Junk in wmf.webrequest.uri_host field - https://phabricator.wikimedia.org/T95836#1859968 (Yurik) Open>Resolved a:Yurik It seems like its much better now, so closing. Thanks!!! [21:36:31] so I want to give you the right syntax [21:38:09] hm, jynus do I start mysql differently than usual? [21:38:13] with wmf-mariadb? [21:38:22] it requires a special config [21:38:22] i see all the stuff is in /opt [21:39:11] the start, yes: /opt/wmf-mariadb10/install && /etc/init.d/mysql start [21:39:23] but tokudb is disabled by default [21:39:50] see puppet/templates/mariadb/analytics.my.cnf.erb [21:41:47] adding this good enough? [21:41:48] plugin-load = ha_tokudb [21:41:52] yes [21:42:07] actually, if you use the production package [21:42:18] the default compression is fine [21:42:39] so you only need to set ENGINE=TokuDB [21:43:08] jynus: when i isntalled the package, it started mysql using my current my.cnf [21:43:11] should I top it and run install? [21:43:28] it looks like it did good things in the logs... [21:43:34] Upgrading MySQL tables if necessary. ... [21:43:44] thne lists all existing tables with OK [21:43:49] should be good? [21:43:51] then you may not have installed our package [21:44:01] but it is ok as far as it is mariadb [21:44:08] oh? [21:44:09] hm [21:44:16] hHMMm [21:44:20] oh i think i installed both [21:44:22] UHmm [21:44:25] :-) [21:44:32] that is on porpuse [21:44:42] yeah Server version: 5.5.46-MariaDB-1ubuntu0.14.04.2 (Ubuntu) [21:44:42] hm [21:44:52] our package installs on /opt so we can upgrade fast [21:45:01] should I uninstall that one? [21:45:04] and use wmf? [21:45:05] yes [21:45:07] ok [21:45:17] uninstall mariadb-server [21:45:36] run /opt/wmf-mariadb10/install [21:46:00] ok, did...it didn't output anything [21:46:01] should it have? [21:46:09] bo [21:46:11] no [21:46:18] ok [21:46:20] then start? [21:46:25] service mysql start? [21:46:33] yes with the init.d script [21:46:43] k [21:46:55] * Couldn't find MySQL manager (/usr/bin/mysqlmanager) or server (/usr/bin/mysqld_safe) [21:47:06] that is probably the config [21:47:12] let me log it and check it myself [21:47:14] k [21:47:17] deployment-eventlogging03 [21:47:18] host? [21:47:24] in deployment-prep project [21:49:30] I think I have my ssh or the bastions misconfigured [21:50:52] maybe yo need access to the project? [21:50:55] i will paste config [21:51:06] that must be it [21:51:27] https://gist.github.com/ottomata/28d85363f3e934e49937 [21:51:34] also, I have 2 usernames, not sure which is the right one for labs [21:52:12] jynus: the one you use in wikitech [21:52:13] yeah, thow away all of that [21:52:46] ottomata: i finds no info as to the mysqlclient and tokudb [21:52:55] ottomata: in python [21:53:19] haha [21:53:20] oh? [21:53:38] nuria: i don't think you will, it is lkely just provided to mysql via the create statement [21:53:47] so, if you can supply options like that with InnoDB [21:53:54] (and if we dont' need other options) [21:54:06] i betcha we can just change the engine to TokuDB in jrm.py and it'll just work [21:54:21] that I can tell you already [21:55:27] look, I do not have the time or energy to fix my config while I am on vacations [21:56:00] just add user = mysql datadir = /srv/sqldata tmpdir = /srv/tmp plugin-load = ha_tokudb [21:56:06] on mysqld [21:56:12] section and it will work [21:56:26] you are on vacations!? [21:56:31] GET OUTTA HERE :) [21:56:37] yes and it is 23 h here [21:56:46] that is why I was trying to reschedule this [21:57:31] but just copy the config from production, the link I sent you [21:57:35] and it will work [21:57:51] the important part is plugin-load = ha_tokudb [21:58:07] and changing all references to the right place [21:59:48] jynus: can't change datadir [21:59:54] the tables are currently in /var/lib/mysql [22:00:30] but it is ok, we don't have to do this now [22:00:39] i will just put old mysql-server back and we can figure out later [22:00:43] yes, either move the tables or the config [22:00:50] k [22:01:40] jynus: on vacation? [22:01:44] jynus: man .. [22:02:07] well, holidays [22:02:36]