[00:01:44] (PS1) Gergő Tisza: Remove czwiki, there is no such thing [analytics/multimedia] - https://gerrit.wikimedia.org/r/148896 [00:02:34] (PS1) Gergő Tisza: Remove czwiki, there is no such thing [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148897 [00:08:50] (PS1) Yuvipanda: Clicking 'Run Query' actually saves the query now! [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 [03:45:51] Analytics / General/Unknown: ULSFO post-move verification - https://bugzilla.wikimedia.org/68199#c9 (Jeff Gage) Ok, by examining router interface statistics with LibreNMS, I have confirmed that when traffic from* ULSFO ceased, traffic from EQIAD increased by a similar amount. * Rather than looking at... [06:30:51] Analytics / Wikimetrics: Story: WikimetricsUser generates report with Target Site breakdown - https://bugzilla.wikimedia.org/68475#c1 (nuria) This requires event login data, which is not public. So this item is contingent on making EL data public. [06:33:21] Analytics / Wikimetrics: Story: WikimetricsUser runs report against all wikis - https://bugzilla.wikimedia.org/68477#c1 (nuria) I think we talked about that on 1st instance this agreggation just plain and simple adds on all numbers. Thus does not deduplicate. We discused this with Erik and it was fine... [10:31:07] Any other link on this? «We experimented with a set up of Riemann for real-time cluster monitoring.» https://www.mediawiki.org/wiki/Analytics/Infrastructure/status#2013-01-monthly [10:33:01] (PS2) Yuvipanda: Add functionality to save and display queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 [10:36:23] (PS3) Yuvipanda: Add functionality to save and display queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 [10:38:22] Analytics / General/Unknown: ULSFO post-move verification - https://bugzilla.wikimedia.org/68199#c10 (Dan Andreescu) Awe-some. Thank you so much Jeff [11:04:50] (PS4) Yuvipanda: Add functionality to save and display queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 [11:22:52] Analytics / General/Unknown: ULSFO post-move verification - https://bugzilla.wikimedia.org/68199#c11 (christian) (In reply to Dan Andreescu from comment #2) > Also found > that only EQIAD hosts had their traffic increase abnormally, [...] I had expected to see amssq47 (esams) being called out, as it p... [11:27:12] Analytics / EventLogging: Delete tofu table from staging database after research is done - https://bugzilla.wikimedia.org/68441 (christian) [11:27:13] Analytics / EventLogging: UniversalLanguageSelector-tofu logging too much data - https://bugzilla.wikimedia.org/67463 (christian) [11:27:42] Analytics / EventLogging: UniversalLanguageSelector-tofu logging too much data - https://bugzilla.wikimedia.org/67463 (christian) [11:27:43] Analytics / EventLogging: Delete tofu table from staging database after research is done - https://bugzilla.wikimedia.org/68441 (christian) [12:30:39] Analytics / Wikimetrics: replication lag may affect recurrent reports - https://bugzilla.wikimedia.org/68507 (Dan Andreescu) NEW p:Unprio s:normal a:None If labsdb has replication lag (which I'm not 100% sure is possible), then recurrent reports might not see all the data for a given day wh... [12:38:40] (PS1) Yuvipanda: Construct user objects in context creation everytime [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148986 [13:27:09] (PS2) Ottomata: Decrease throttle for webrequest log verification [analytics/refinery] - https://gerrit.wikimedia.org/r/148700 (owner: QChris) [13:27:14] (CR) Ottomata: [C: 2 V: 2] Decrease throttle for webrequest log verification [analytics/refinery] - https://gerrit.wikimedia.org/r/148700 (owner: QChris) [13:29:38] (PS2) Ottomata: Rename script to generate sequence statistics [analytics/refinery] - https://gerrit.wikimedia.org/r/148863 (owner: QChris) [13:29:43] (CR) Ottomata: [C: 2 V: 2] Rename script to generate sequence statistics [analytics/refinery] - https://gerrit.wikimedia.org/r/148863 (owner: QChris) [13:39:21] qchris, you got it :) [13:49:45] thanks ottomata! [13:51:42] [2014-07-24 10:31:08] Any other link on this? �We experimented with a set up of Riemann for real-time cluster monitoring.� https://www.mediawiki.org/wiki/Analytics/Infrastructure/status#2013-01-monthly [13:51:46] milimetric, ottomata: ^ do you know more about that? [13:52:15] yeah, I remember David set it up [13:52:25] I'm not sure what came of it, but it was very cool UI-wise [13:52:49] Ok. Thanks. [13:52:57] Nemo_bis: You are reading along? [13:53:26] qchris: do you have any idea how to iptables-restore this: http://paste.ubuntu.com/7847723/ [13:53:41] * qchris looks [13:54:24] Just running the iptables commands does not work? [13:54:29] Missing DNAT module? [13:54:48] Or what's the issue? [13:57:40] milimetric: do you have those rules installed somwhere? [13:57:41] you can do [13:57:42] iptables-save [13:57:49] to get an output of the commands to run [13:59:07] ottomata: can't do -save, I need sudo on tools [13:59:11] so someone pasted taht for me [13:59:54] thanks milimetric [14:00:54] milimetric: iptables -t nat -I OUTPUT -p tcp --destination s1.labsdb --dport mysql -j DNAT --to-destination 10.64.20.12:3306 [14:01:02] milimetric: Does that get you started? [14:01:15] (It should give you line 9 of the dpaste) [14:07:53] (PS1) Yuvipanda: Setup QueryRun model and API endpoints [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148999 [14:10:07] Analytics / General/Unknown: ULSFO post-move verification - https://bugzilla.wikimedia.org/68199 (Dan Andreescu) a:Dan Andreescu [14:17:40] (CR) Gilles: [C: 2 V: 2] Remove czwiki, there is no such thing [analytics/multimedia] - https://gerrit.wikimedia.org/r/148896 (owner: Gergő Tisza) [14:17:48] (CR) Gilles: [C: 2 V: 2] Remove czwiki, there is no such thing [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148897 (owner: Gergő Tisza) [14:25:35] (CR) Gilles: [C: 2 V: 2] Use correct DB for optout queries [analytics/multimedia] - https://gerrit.wikimedia.org/r/148838 (owner: Gergő Tisza) [14:26:31] (CR) Gilles: [C: 2 V: 2] Record optout stats for users with 1+ edits and 100+ edits [analytics/multimedia] - https://gerrit.wikimedia.org/r/148842 (owner: Gergő Tisza) [14:31:48] (PS1) QChris: Take advantage of CNAMEs for namenode and resourcemanager [analytics/refinery] - https://gerrit.wikimedia.org/r/149007 [14:32:14] ottomata: ^ [14:32:39] u so fast [14:33:13] (CR) QChris: Take advantage of CNAMEs for namenode and resourcemanager (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/149007 (owner: QChris) [14:33:19] qchris, is hadoop_master used anywhere anymore? [14:33:41] I do not know. [14:33:43] I hope not. [14:33:45] haha [14:33:49] will pull and check [14:34:05] But I wanted to haev something that uses cnames, so you can deploy. [14:34:45] I could not grep another use of hadoop_master. [14:34:52] You prefer we get rid of it? [14:35:13] if it isn't used, ja [14:35:40] hm, workflow.properties? [14:35:49] did you delete that file? [14:36:16] (PS2) QChris: Take advantage of CNAMEs for namenode and resourcemanager [analytics/refinery] - https://gerrit.wikimedia.org/r/149007 [14:36:34] No, it became bundle.properties. [14:37:02] Just use that and override the bundle property with a workflow property [14:37:13] (Do not forget to set webrequest_source) [14:37:25] Meeting is starting ... [14:39:10] k cool [14:40:32] (CR) Ottomata: [C: 2 V: 2] Take advantage of CNAMEs for namenode and resourcemanager [analytics/refinery] - https://gerrit.wikimedia.org/r/149007 (owner: QChris) [14:42:34] (PS1) Gilles: Make the generate SQL folder if needed [analytics/multimedia] - https://gerrit.wikimedia.org/r/149014 [14:42:57] (CR) Gilles: [C: 2 V: 2] "Self-merging trivial change" [analytics/multimedia] - https://gerrit.wikimedia.org/r/149014 (owner: Gilles) [14:49:06] Analytics / Wikimetrics: EEVSUser downloads report with correct Http Cache Headers - https://bugzilla.wikimedia.org/68445 (Kevin Leduc) [14:57:39] (PS1) Gilles: Fix global tsp by generating it like the others [analytics/multimedia] - https://gerrit.wikimedia.org/r/149017 [14:58:08] (CR) Gilles: [C: 2 V: 2] "Self-merging, I'm saving time by working directly on the machine and trying to make it all work" [analytics/multimedia] - https://gerrit.wikimedia.org/r/149017 (owner: Gilles) [15:01:37] (PS1) Gilles: Make global sql work [analytics/multimedia] - https://gerrit.wikimedia.org/r/149018 [15:01:48] (CR) Gilles: [C: 2 V: 2] Make global sql work [analytics/multimedia] - https://gerrit.wikimedia.org/r/149018 (owner: Gilles) [15:04:40] (PS1) Gilles: Revert "Make global sql work" [analytics/multimedia] - https://gerrit.wikimedia.org/r/149022 [15:04:49] (CR) Gilles: [C: 2 V: 2] Revert "Make global sql work" [analytics/multimedia] - https://gerrit.wikimedia.org/r/149022 (owner: Gilles) [15:04:54] (PS1) Gilles: Revert "Fix global tsp by generating it like the others" [analytics/multimedia] - https://gerrit.wikimedia.org/r/149023 [15:05:02] (CR) Gilles: [C: 2 V: 2] Revert "Fix global tsp by generating it like the others" [analytics/multimedia] - https://gerrit.wikimedia.org/r/149023 (owner: Gilles) [15:07:58] (PS1) Gilles: Copy over global.sql [analytics/multimedia] - https://gerrit.wikimedia.org/r/149024 [15:08:40] (PS2) Gilles: Copy over global.sql [analytics/multimedia] - https://gerrit.wikimedia.org/r/149024 [15:09:01] (CR) Gilles: [C: 2 V: 2] Copy over global.sql [analytics/multimedia] - https://gerrit.wikimedia.org/r/149024 (owner: Gilles) [15:22:53] qchris_meeting: i cannot kill oozie job either?! :( [15:29:51] (PS1) Gilles: Point optout datasource to the correct server [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149026 [15:30:18] (CR) Gilles: [C: 2 V: 2] Point optout datasource to the correct server [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149026 (owner: Gilles) [15:52:59] (PS1) Gilles: Run optout global.sql last and fix queries [analytics/multimedia] - https://gerrit.wikimedia.org/r/149032 [15:53:11] (CR) Gilles: [C: 2 V: 2] Run optout global.sql last and fix queries [analytics/multimedia] - https://gerrit.wikimedia.org/r/149032 (owner: Gilles) [16:03:20] (PS1) Gilles: Fix trailing comma in JSON [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149034 [16:03:33] (CR) Gilles: [C: 2 V: 2] Fix trailing comma in JSON [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149034 (owner: Gilles) [16:14:39] (PS1) Gilles: Fix datasource field order [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149037 [16:14:51] (CR) Gilles: [C: 2 V: 2] Fix datasource field order [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149037 (owner: Gilles) [16:17:52] Analytics / Wikimetrics: Story: EEVSUser adds 'Edits' metric - https://bugzilla.wikimedia.org/68352 (Kevin Leduc) [16:18:12] oh why nobody ask for jenkins jobs :D [16:20:01] (PS1) Gilles: Add very active users to tsvs [analytics/multimedia] - https://gerrit.wikimedia.org/r/149039 [16:20:14] (CR) Gilles: [C: 2 V: 2] Add very active users to tsvs [analytics/multimedia] - https://gerrit.wikimedia.org/r/149039 (owner: Gilles) [16:23:33] hashar: I have a patch for one of my things! [16:23:35] moment [16:25:12] hashar: https://gerrit.wikimedia.org/r/#/c/148361/ and https://gerrit.wikimedia.org/r/#/c/148362/ [16:30:34] YuviPanda: :D [16:30:49] goood job [16:31:55] (PS1) Gilles: Add very active users to graphs [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149040 [16:32:04] YuviPanda: or you could use tox :] [16:32:06] (CR) Gilles: [C: 2 V: 2] Add very active users to graphs [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149040 (owner: Gilles) [16:34:27] hashar: :D legoktm did it [16:34:51] and now Jenkins is screwed again [16:34:53] pfff [16:36:26] [travis-ci] wikimedia/mediawiki-extensions-EventLogging#230 (wmf/1.24wmf15 - d58240a : Reedy): The build passed. [16:36:26] [travis-ci] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/d58240a2efd4 [16:36:26] [travis-ci] Build details : http://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/30766813 [16:46:00] YuviPanda: jobs are enabled :D [16:46:06] hashar: w00t. ty [16:46:47] I will have to look at that travis build as well :D [16:47:31] (PS7) Gilles: Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza) [16:48:26] I am out [16:52:11] (PS8) Gilles: Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza) [16:52:52] (CR) Gilles: [C: 2 V: 2] Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza) [16:54:07] (CR) Yuvipanda: [C: 2] Add functionality to save and display queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 (owner: Yuvipanda) [16:54:10] (CR) jenkins-bot: [V: -1] Add functionality to save and display queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 (owner: Yuvipanda) [16:55:20] (PS1) Gilles: Create duration sql folder when needed [analytics/multimedia] - https://gerrit.wikimedia.org/r/149041 [16:55:36] (CR) Gilles: [C: 2 V: 2] Create duration sql folder when needed [analytics/multimedia] - https://gerrit.wikimedia.org/r/149041 (owner: Gilles) [16:57:20] (PS1) Yuvipanda: Add tox.ini [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149042 [16:57:41] (PS1) Gilles: Fix %wiki% clause in duration template [analytics/multimedia] - https://gerrit.wikimedia.org/r/149043 [16:57:53] (CR) Gilles: [C: 2 V: 2] Fix %wiki% clause in duration template [analytics/multimedia] - https://gerrit.wikimedia.org/r/149043 (owner: Gilles) [16:59:16] (PS2) Yuvipanda: Add tox.ini [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149042 [16:59:30] (CR) Yuvipanda: [C: 2] Add tox.ini [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149042 (owner: Yuvipanda) [16:59:32] (Merged) jenkins-bot: Add tox.ini [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149042 (owner: Yuvipanda) [17:00:02] (PS5) Yuvipanda: Add functionality to save and display queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 [17:00:04] (CR) jenkins-bot: [V: -1] Add functionality to save and display queries [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 (owner: Yuvipanda) [17:00:59] (CR) Yuvipanda: [V: 2] "Stupid Jenkins not picking up tox.ini" [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 (owner: Yuvipanda) [17:01:30] (CR) Yuvipanda: "BAH NOW I CAN'T MERGE. STUPID JENKINS." [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 (owner: Yuvipanda) [17:05:24] Analytics / General/Unknown: ULSFO post-move verification - https://bugzilla.wikimedia.org/68199#c12 (Jeff Gage) Hi, We should definitely trust the commits over my information source, the Server Admin Log whose events are manually input: https://wikitech.wikimedia.org/wiki/SAL 22:42 mark: Enabling PA... [17:12:06] Analytics / Refinery: Story: Admin has duplicate monitoring in Icinga - https://bugzilla.wikimedia.org/67128 (Dan Andreescu) PATC>RESO/FIX [17:14:50] Analytics / General/Unknown: ULSFO post-move verification - https://bugzilla.wikimedia.org/68199 (Dan Andreescu) [17:15:21] Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694 (Dan Andreescu) [17:16:35] Analytics / Wikimetrics: Story: Researcher has prototype for wikimania - https://bugzilla.wikimedia.org/68516 (Dan Andreescu) [17:22:05] Analytics / Wikimetrics: Story: Researcher has prototype for wikimania - https://bugzilla.wikimedia.org/68516 (Dan Andreescu) [17:33:33] (PS2) Yuvipanda: Construct user objects in context creation everytime [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148986 [17:33:35] (CR) jenkins-bot: [V: -1] Construct user objects in context creation everytime [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148986 (owner: Yuvipanda) [17:33:45] (PS2) Yuvipanda: Setup QueryRun model and API endpoints [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148999 [17:33:47] (CR) jenkins-bot: [V: -1] Setup QueryRun model and API endpoints [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148999 (owner: Yuvipanda) [17:44:22] Analytics / General/Unknown: Use zero.wikimedia.org's API to get carrier configuration - https://bugzilla.wikimedia.org/68519 (christian) NEW p:Unprio s:normal a:None This would allow the Wikipedia Zero team to stop maintaining the configuration at different places (like meta). An example... [17:46:33] https://identi.ca/mlinksva/note/w6qCCMF1T2qvlFxlKVcdzg [17:46:42] relavent to your interests [17:48:06] Analytics / General/Unknown: Use zero.wikimedia.org's API to get carrier configuration - https://bugzilla.wikimedia.org/68519 (christian) [17:49:57] Meh ... who uses Google Analytics? :-/ [18:00:03] (PS3) Gilles: Track loading time for MediaViewer and the file page [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148024 (owner: Gergő Tisza) [18:07:21] (PS4) Gilles: Track loading time for MediaViewer and the file page [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148024 (owner: Gergő Tisza) [18:07:38] (CR) Gilles: [C: 2 V: 2] Track loading time for MediaViewer and the file page [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148024 (owner: Gergő Tisza) [18:12:52] Analytics / Wikimetrics: Deal with scheduled recurrent reports when cohorts are deleted - https://bugzilla.wikimedia.org/68520 (nuria) NEW p:Unprio s:normal a:None Deal with scheduled recurrent reports when cohorts are deleted. When a cohort is deleted all recurrent reports scheduled to tha... [18:21:14] (PS1) Gilles: Restore graph that was incorrectly deleted + git ignored [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149072 [18:21:27] (CR) Gilles: [C: 2 V: 2] Restore graph that was incorrectly deleted + git ignored [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149072 (owner: Gilles) [18:32:03] (PS3) Yuvipanda: Setup QueryRun model and API endpoints [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148999 [18:32:05] (PS1) Yuvipanda: Expand code area to fit content [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149078 [18:32:09] (CR) jenkins-bot: [V: -1] Setup QueryRun model and API endpoints [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148999 (owner: Yuvipanda) [18:33:02] (PS1) Gilles: Restore global graph that was deleted/git ignored accidentally [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149079 [18:33:13] (CR) Gilles: [C: 2 V: 2] Restore global graph that was deleted/git ignored accidentally [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149079 (owner: Gilles) [18:45:21] finally.. a breakthrough! \o/ [19:25:43] YuviPanda: well analytics/quarry/web jobs are not passing :/ [19:26:00] YuviPanda: should we make the jobs non voting ? [19:26:03] hashar: yeah, it's not taking into account my tox.ini [19:26:08] ah no [19:26:10] hashar: can we switch off from flake/pep8 to tox? [19:26:34] hashar: it passes locally, and the errors reported are line length errors that I've turned off [19:26:53] your tox.ini only has a flake8 section [19:27:01] but yeah lets move to tox [19:27:09] hashar: yeah, no tests yet :) [19:28:19] https://gerrit.wikimedia.org/r/149094 :D [19:29:03] hashar: :D [19:30:18] hashar: oh wow, didn't realize I had +2 there [19:30:34] YuviPanda: well the job need to be manually deployed [19:30:37] and the diff reviewed :D [19:30:55] hashar: can I bribe you into doing that? :) [19:32:49] did :-) [19:32:53] hashar: w00t [19:32:54] zuul there https://gerrit.wikimedia.org/r/149095 [19:34:07] (CR) Hashar: "recheck" [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 (owner: Yuvipanda) [19:34:30] stupid jenkins [19:35:16] (CR) Hashar: "recheck" [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148898 (owner: Yuvipanda) [19:35:55] (CR) Hashar: "recheck" [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148986 (owner: Yuvipanda) [19:36:04] (CR) Hashar: "recheck" [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148999 (owner: Yuvipanda) [19:36:11] (CR) Hashar: "recheck" [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149078 (owner: Yuvipanda) [19:36:24] all Verified +2 by jenkins [19:36:40] hashar: w00t, ty [19:36:45] YuviPanda: and I really really recommend to start writing tests right now [19:37:06] hashar: yeah, I've been writing code to make it easy enough to test, will add soon [19:37:19] hashar: the project is only a few days old at this point, should have some tests at least when it's a week or so old [19:37:29] what is Flask compared to Django ? [19:37:34] is that some kind of micro framework? [19:37:47] hashar: yeah [19:38:07] hashar: a lot popular over the last few years :D much simpler [19:38:07] how the hell do you find time to learn of those frameworks/languages ? :D [19:38:36] hashar: :D I was using it 2-3 years ago before I started doing Android stuff. [19:39:21] anyway when you want test added [19:39:26] just add a toxenv like py27 [19:39:43] + a py27 job in the jenkins job template. [19:39:50] (CR) Yuvipanda: [C: 2] Construct user objects in context creation everytime [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148986 (owner: Yuvipanda) [19:39:54] the job will need to be created [19:40:01] then add the trigger in zuul :D [19:40:13] hashar: yeah :D also needs to be py3 compatible. Right now a library isn't, need to fix that as well [19:40:18] (Merged) jenkins-bot: Construct user objects in context creation everytime [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148986 (owner: Yuvipanda) [19:40:42] Analytics / Wikimetrics: Deal with scheduled recurrent reports when cohorts are deleted - https://bugzilla.wikimedia.org/68520#c1 (Dan Andreescu) For now, these were causing lots of problems so I set recurrent=0 on them: update report set recurrent=0 where recurrent=1 and (parameters like '%"id": 834%... [19:41:08] (PS4) Yuvipanda: Setup QueryRun model and API endpoints [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148999 [19:41:14] (CR) Yuvipanda: [C: 2] Setup QueryRun model and API endpoints [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148999 (owner: Yuvipanda) [19:41:17] YuviPanda: jenkins slaves in labs could get python 3.2 installed [19:41:21] (Merged) jenkins-bot: Setup QueryRun model and API endpoints [analytics/quarry/web] - https://gerrit.wikimedia.org/r/148999 (owner: Yuvipanda) [19:41:25] 3.3 is not in our apt.wikimedia.org unfortunately [19:41:29] hashar: needs 3.4, so needs trusty [19:41:36] (PS2) Yuvipanda: Expand code area to fit content [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149078 [19:41:42] (CR) Yuvipanda: [C: 2] Expand code area to fit content [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149078 (owner: Yuvipanda) [19:41:47] (Merged) jenkins-bot: Expand code area to fit content [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149078 (owner: Yuvipanda) [19:55:10] (PS1) Gergő Tisza: Take sampling factor into account for duration data [analytics/multimedia] - https://gerrit.wikimedia.org/r/149101 [19:56:29] (CR) Gilles: [C: 2 V: 2] Take sampling factor into account for duration data [analytics/multimedia] - https://gerrit.wikimedia.org/r/149101 (owner: Gergő Tisza) [19:57:31] (PS1) Gergő Tisza: Take sampling factor into account in duration data [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149102 [20:01:16] (CR) Gilles: [C: 2 V: 2] Take sampling factor into account in duration data [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149102 (owner: Gergő Tisza) [20:02:58] (PS1) Ottomata: Alter names of oozie jobs to make it easier to identify in Oozie UIs. [analytics/refinery] - https://gerrit.wikimedia.org/r/149107 [20:05:08] (PS1) Gergő Tisza: More informative description for duration graph [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149109 [20:08:52] hey ottomata. [20:09:07] Got a sec for me to ask you about some stat3 issues? [20:09:14] sure [20:09:51] I pull data between servers a lot. That means I need my creds on a server to pull and push. I've been using ssh-agent for this (On the sec of leslie carr). [20:10:02] This suddenly stopped working a few days ago. [20:10:41] Interestingly, I can use my ssh-agent on stat1002, but I can't on stat1003. [20:11:16] No config changes on my end, though I did try all sorts of ways to force the agent to get forwarded once I noticed the problem. [20:11:42] Buh.. hmm... suddenly it's working. [20:12:07] That's the longest duration ask-and-then-it-suddenly-works I've ever experienced. [20:12:16] n/m I guess! [20:14:30] anyone to help me with a bit of SQL? [20:14:39] haah [20:14:41] i didn't do it! [20:14:44] not related to prod dbs, just am trying to do a join between two tables and failing miserably [20:14:46] halfak: [20:14:47] but [20:14:49] there is a better way [20:15:02] the stat machines have an rsync module that allows for rsyncing of anything in the /a directories between each other [20:15:12] YuviPanda, I can help. [20:15:26] hashar: https://dpaste.de/uheZ [20:15:46] err [20:15:47] halfak: ^ [20:15:50] halfak: that gives me 'ERROR 1054 (42S22): Unknown column 'query.id' in 'on clause'' [20:15:50] ottomata, I need to be able to move data between more than just the stats machines. [20:15:55] where else? [20:16:09] https://github.com/wikimedia/analytics-quarry-web/blob/master/tables.sql is the schema [20:16:40] labs machines. Hmm.. Sometimes my local server. I develop SQL and other server code on stat3. it's nice to be able to interact directly with github/gerrit. [20:17:39] YuviPanda, first, you're naming is terrifying. query, query_revision and query_run are all tables? [20:17:41] me too, i can't help you wiht that then [20:17:57] but, you shouldn't need ssh agents to copy files between your local machine and any remote machine [20:18:12] Local server :P [20:18:18] sure, whatever [20:18:21] if your ssh key is there [20:18:29] i guess it snot...? [20:18:38] halfak: found the error, I was using query.id when it should've been query_revision.id [20:18:41] Ahh. I don't like to put it there since I don't encrypt the hard drives on the server (spinning disks) [20:19:05] YuviPanda, cool [20:19:05] halfak: and yes, I haven't had to design schemas in like 3 years :D What should have they been? [20:19:10] I'd probably alias them for the query [20:19:26] YuviPanda, Not sure. I'd need to look at the table defs [20:19:38] halfak: https://github.com/wikimedia/analytics-quarry-web/blob/master/tables.sql [20:20:06] OH! The name are more reasonable than I thought. [20:20:40] halfak: :D [20:21:39] halfak: I'd probably denormalize a bit in the future if the performance is terrible, but this seems adequete for now... [20:22:08] +1 for early normalization. Denormalization is for analytics. [20:22:29] All looks solid to me. [20:24:25] halfak: yay :) [20:30:36] aaaaaaaaaaah [20:31:00] aaaaaaaaah! [20:31:10] ottomata: what is processorcount? [20:31:11] https://github.com/wikimedia/operations-puppet-wikimetrics/blob/master/manifests/init.pp#L87 [20:35:50] milimetric: its the number of processors on the node [20:36:18] milimetric: run [20:36:19] facter [20:36:20] on any node [20:36:23] :( [20:36:26] to see all default available facts [20:36:28] what'sup? [20:36:35] so, in our case on wikimetrics prod, that value is just the empty string [20:36:44] ? weird, really? [20:36:49] and either way, it doesn't make any sense there, that should be hardcoded to like 100 or something [20:37:01] something like debug ? 16 : 100 [20:37:02] k, change it! [20:37:09] but, there was no reason that was there? [20:37:17] seems kind of out there to be a random change... [20:37:33] was it recently changed? [20:37:56] it shouldn't be blank [20:37:57] for sure [20:38:01] [@wikimetrics1:~] $ facter processorcount [20:38:01] 4 [20:38:23] YuviPanda: https://dpaste.de/uheZ <--- unknown language to me :-D [20:38:33] YuviPanda: I can't remember the last time I had to look/write sql [20:38:34] hashar: yeah, tab complete confusion, sorry [20:38:42] YuviPanda: teasing you :D [20:38:50] ah [20:38:51] milimetric: [20:38:56] there is a typo in the yaml [20:38:58] template [20:38:59] CELERYD_CONCURRENCY : <% @celery_concurrency %> [20:39:00] should be [20:39:03] CELERYD_CONCURRENCY : <%= @celery_concurrency %> [20:39:10] fixing... [20:39:14] wait ottomata [20:39:17] ok [20:39:26] it'll be blank for anything [20:39:28] it shouldn't be processorcount though, in any case [20:51:12] milimetric: merged. [20:51:30] thanks, are you going to be around for a bit? [20:54:10] oh, actually it's fine I'll just checkout any changes from gerrit, no need to merge [21:04:14] ok cool, yeah, getting ready to head out soon [21:18:04] Analytics / Wikimetrics: QueuePool pool_size limit should be configurable - https://bugzilla.wikimedia.org/68534 (Dan Andreescu) NEW p:Unprio s:normal a:None Error message that led to discovery of the problem: Problem creating child report: QueuePool limit of size 5 overflow 10 reached, co... [21:18:14] Analytics / Wikimetrics: QueuePool pool_size limit should be configurable - https://bugzilla.wikimedia.org/68534 (Dan Andreescu) [21:20:53] (PS1) Milimetric: Improve connection pool configurability [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/149123 (https://bugzilla.wikimedia.org/68534) [21:42:19] (CR) Gergő Tisza: [C: 2] "Self-merge, trivial text change" [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149109 (owner: Gergő Tisza) [21:45:42] (PS6) Terrrydactyl: [WIP] Add ability to global query a user's wikis [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129858 [21:45:44] (CR) jenkins-bot: [V: -1] [WIP] Add ability to global query a user's wikis [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129858 (owner: Terrrydactyl) [21:47:29] (CR) Gergő Tisza: [V: 2] More informative description for duration graph [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149109 (owner: Gergő Tisza) [21:56:12] (PS1) Gergő Tisza: Clarify description of performance Selenium test [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149172 [21:56:40] (CR) Gergő Tisza: [C: 2 V: 2] "Trivial text change." [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/149172 (owner: Gergő Tisza) [22:14:58] (PS1) Yuvipanda: Display all query runs ever in page [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149176 [22:15:00] (CR) jenkins-bot: [V: -1] Display all query runs ever in page [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149176 (owner: Yuvipanda) [22:15:24] (PS1) John F. Lewis: Change header to Quarry [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149178 [22:17:00] (CR) Yuvipanda: [C: 2] "YAY" [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149178 (owner: John F. Lewis) [22:17:02] (Merged) jenkins-bot: Change header to Quarry [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149178 (owner: John F. Lewis) [22:18:30] (PS2) Yuvipanda: Display all query runs ever in page [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149176 [22:20:53] anyone know if analytics1027 has been puppet disabled on purpose? [22:21:01] if not, i guess i'll renable it [22:21:11] (CR) Legoktm: [C: 2] Display all query runs ever in page [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149176 (owner: Yuvipanda) [22:21:13] there's a bug that does ot sometimes [22:21:16] (Merged) jenkins-bot: Display all query runs ever in page [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149176 (owner: Yuvipanda) [22:42:07] (PS1) Yuvipanda: Set min-height on the code editor [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149182 [22:42:09] (CR) jenkins-bot: [V: -1] Set min-height on the code editor [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149182 (owner: Yuvipanda) [22:42:29] (PS2) Yuvipanda: Set min-height on the code editor [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149182 [22:42:36] (CR) Yuvipanda: [C: 2] Set min-height on the code editor [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149182 (owner: Yuvipanda) [22:42:41] (Merged) jenkins-bot: Set min-height on the code editor [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149182 (owner: Yuvipanda) [23:31:21] (PS1) Yuvipanda: Added full bootstrap release [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149188 [23:31:37] (PS2) Yuvipanda: Added full bootstrap release [analytics/quarry/web] - https://gerrit.wikimedia.org/r/149188