[00:15:36] <wikibugs>	 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10aezell) This can't be worked on in the CommTech sprint until, and if, we get the beacon set up by the Analytics team.
[00:15:50] <wikibugs>	 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10aezell)
[05:18:17] <wikibugs>	 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10Nuria) Tracking code:  <!-- Matomo --> <script type="text/javascript">   var _paq = _paq || [];   /* tracker methods like...
[06:09:33] <wikibugs>	 10Analytics, 10Community-Tech, 10SVG Translate Tool, 10Community-Tech-Sprint: Integrate Piwik with SVG Translate to keep track of metrics - https://phabricator.wikimedia.org/T215478 (10Samwilson) a:03Samwilson PR: https://github.com/wikimedia/svgtranslate/pull/81
[08:05:01] <wikibugs>	 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10elukey) Via HPE CLI I tried to flip /map1/config1/oemHPE_ipmi_dcmi_overlan_enable=yes but didn't work afaics..
[08:15:24] <wikibugs>	 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['labsdb1012.eqiad.wmnet'] ` The log...
[08:29:58] <wikibugs>	 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10elukey) The above setting seems to have done the trick, now wmf-auto-reimage works.. I got this:  ` 08:27:57 | labsdb1012.eqiad.wmnet | WARNI...
[08:31:50] <wikibugs>	 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['labsdb1012.eqiad.wmnet'] `  and were **ALL** successful.
[08:41:44] <wikibugs>	 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10elukey) a:05Cmjohnson→03elukey
[09:14:28] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Operations, 10Product-Analytics, 10Patch-For-Review: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) dbstore1002 just crashed: ` Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysql...
[09:14:58] <wikibugs>	 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: [EventLogging Sanitization] Enable older-than-90-day purging of unsanitized EL database (event) in Hive - https://phabricator.wikimedia.org/T209503 (10GoranSMilovanovic) @mforns Hi. No, I do not own WMDEBannerEvents of WMDEBannerSizeIssue, but...
[12:40:14] * elukey lunch!
[14:10:51] <joal>	 Heya elukey - Just sent a message, taking today off (sorry for the late notice)
[14:11:42] <elukey>	 joal: au revoir! 
[14:12:00] <joal>	 :)
[14:12:23] <joal>	 all good for you elukey ?
[14:12:56] <elukey>	 yep! labsdb1012 is finally ready to get its puppet config (next week), I am currently working on puppet 
[14:12:59] <elukey>	 sigh
[14:13:29] <joal>	 elukey: you should advertise yourself as a DBA ;)
[14:16:04] <elukey>	 suuuuure
[14:59:59] <wikibugs>	 10Analytics, 10User-Elukey: Enable Security (stronger authentication and data encryption) for the Analytics Hadoop cluster and its dependent services - https://phabricator.wikimedia.org/T211836 (10elukey)
[15:04:01] <wikibugs>	 (03PS1) 10Ladsgroup: Fix active_users.php by doing most of the work in code [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493701 (https://phabricator.wikimedia.org/T213894)
[15:04:10] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Fix active_users.php by doing most of the work in code [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493701 (https://phabricator.wikimedia.org/T213894) (owner: 10Ladsgroup)
[15:08:44] <wikibugs>	 10Analytics, 10User-Elukey: Enable encryption and authentication for TLS-based Hadoop services - https://phabricator.wikimedia.org/T217412 (10elukey) p:05Triage→03High
[15:08:52] <wikibugs>	 (03CR) 10Ladsgroup: "recheck" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493701 (https://phabricator.wikimedia.org/T213894) (owner: 10Ladsgroup)
[15:09:13] <bearloga>	 joal: CR for you :) https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/493424/
[15:10:44] <wikibugs>	 (03CR) 10WMDE-leszek: [C: 04-1] "I am no knowledged how it was broken before the change (i.e. what is the need for "fix"), also could add the usual concern about some solu" (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493701 (https://phabricator.wikimedia.org/T213894) (owner: 10Ladsgroup)
[15:13:33] <wikibugs>	 (03CR) 10Ladsgroup: "Yes, it was broken because dbstore1002 is read-only for more than a week now :D This has been broken since then." (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493701 (https://phabricator.wikimedia.org/T213894) (owner: 10Ladsgroup)
[15:16:09] <wikibugs>	 (03PS2) 10Ladsgroup: Fix active_users.php by doing most of the work in code [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493701 (https://phabricator.wikimedia.org/T213894)
[15:18:38] <wikibugs>	 (03CR) 10WMDE-leszek: [C: 03+2] Fix active_users.php by doing most of the work in code [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493701 (https://phabricator.wikimedia.org/T213894) (owner: 10Ladsgroup)
[15:18:46] <wikibugs>	 (03Merged) 10jenkins-bot: Fix active_users.php by doing most of the work in code [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493701 (https://phabricator.wikimedia.org/T213894) (owner: 10Ladsgroup)
[15:21:29] <wikibugs>	 (03PS1) 10Ladsgroup: Fix active_users.php by doing most of the work in code [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493706 (https://phabricator.wikimedia.org/T213894)
[15:21:37] <wikibugs>	 (03CR) 10Ladsgroup: [C: 03+2] Fix active_users.php by doing most of the work in code [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493706 (https://phabricator.wikimedia.org/T213894) (owner: 10Ladsgroup)
[15:22:00] <wikibugs>	 (03Merged) 10jenkins-bot: Fix active_users.php by doing most of the work in code [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/493706 (https://phabricator.wikimedia.org/T213894) (owner: 10Ladsgroup)
[15:29:48] <wikibugs>	 (03CR) 10Nuria: Update whitelisting for Android-related schemas (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493424 (https://phabricator.wikimedia.org/T209087) (owner: 10Bearloga)
[15:31:09] <wikibugs>	 (03PS2) 10Bearloga: Update whitelisting for Android-related schemas [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493424 (https://phabricator.wikimedia.org/T209087)
[15:31:37] <wikibugs>	 (03CR) 10Bearloga: "Good point!" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493424 (https://phabricator.wikimedia.org/T209087) (owner: 10Bearloga)
[15:59:04] <Amir1>	 elukey: hey, around?
[16:01:10] <elukey>	 Amir1: o/ I am yes (but in a meeting soon so I might lag in answering)
[16:01:39] <Amir1>	 elukey: just wanted to say looking at the logs, writing to the staging node fails all the time
[16:01:52] <Amir1>	 maybe the user doesn't have the rights?
[16:01:59] <elukey>	 ouch
[16:02:11] <elukey>	 do you have a specific failure?
[16:02:15] <elukey>	 I mean error message
[16:02:20] <elukey>	 also, what user are you using?
[16:02:24] <elukey>	 'research' ?
[16:02:28] <Amir1>	 the user can be found at /etc/mysql/conf.d/research-wmde-client.cnf
[16:02:36] <elukey>	 ahhh lemme check
[16:02:40] <Amir1>	 (I don't know I know it reads the user from this file)
[16:02:56] <Amir1>	 The error is not output
[16:03:05] <Amir1>	 it's just "2019-03-01 09:58:00 betafeature-counts FAILED: TRUNCATE TABLE staging.wmde_analytics_betafeature_users_today"
[16:03:24] <Amir1>	 While I was able to do exact same query when I connected to staging
[16:03:27] <elukey>	 yeah so you are using the research user
[16:24:40] <Amir1>	 elukey: so is it easily fixable?
[16:29:14] <elukey>	 Amir1: sorry still in a meeting, going to check in ~30 mins ;(
[16:29:32] <Amir1>	 no worries. Next run is 03 am UTC :D
[16:57:53] <elukey>	 Amir1: can you tell me a bit more about what statement gets the error? Also what client you are using etc..
[16:58:11] <elukey>	 I am not super expert so I'd need a bit more details to figure out what's wrong :)
[16:58:20] <elukey>	 and I think that the dbas are already in weekend mode
[16:58:23] <Amir1>	 elukey: e.g. it's 'SELECT COUNT(*) AS count, feature FROM staging.wmde_analytics_betafeature_users_today GROUP BY feature'
[16:58:30] <Amir1>	 but basically everything fails
[16:59:31] <Amir1>	 or 'TRUNCATE TABLE staging.wmde_analytics_betafeature_users_today' 
[16:59:34] <Amir1>	 or 'INSERT INTO staging.wmde_analytics_betafeature_users_yesterday ( user_name, feature ) SELECT user_name, feature FROM staging.wmde_analytics_betafeature_users_today'
[16:59:40] <Amir1>	 the client is php PDO
[17:00:17] <Amir1>	 the code is https://github.com/wikimedia/analytics-wmde-scripts/blob/master/lib/WikimediaDb.php#L53
[17:00:27] <Amir1>	 the staging fail only
[17:00:43] <Amir1>	 elukey: Sorry for the flood of information ^
[17:00:48] <elukey>	 nono thank you :)
[17:00:58] <mforns>	 a-team standuppppp
[17:01:00] <elukey>	 so the first query (select count(*)) returns a empty result
[17:01:22] <Amir1>	 yeah but in code it returns error I think
[17:01:32] <Amir1>	 not empty
[17:02:10] <elukey>	 yep yep I was wondering if that was causing the error for some reason
[17:02:50] <Amir1>	 I tried those with my connection (mysql -h staging-db-analytics.eqiad.wmnet -P 3350 -A) and it runs just fine
[17:03:18] <Amir1>	 the only thing that makes sense to me is user rights 
[17:03:28] <Amir1>	 (I was also able to do writes as well)
[17:08:10] <elukey>	 Amir1: can you retry with the credentials in the stat cnf conf?
[17:10:19] <Amir1>	 that needs a puppet change + making sure the user has the read access there. I can try running it directly though
[17:10:23] <Amir1>	 let me try that
[17:12:24] <elukey>	 ah yes sorry I meant manually
[17:12:39] <Amir1>	 let me try
[17:12:59] <Amir1>	 going to try with '/etc/mysql/conf.d/analytics-research-client.cnf'
[17:13:49] <Amir1>	 the big problem is that it's pretty slow :(
[17:17:32] <elukey>	 Amir1: just to be sure - did you also try, manually, with the wmde cnf file conf or user in it? 
[17:18:12] <Amir1>	 no 
[17:18:33] <Amir1>	 basically the reason is that I get access to file error 
[17:20:05] <elukey>	 ah ok understood
[17:22:07] <elukey>	 so you are in analytics-wmde-users, but the file is owned by root:analytics-wmde
[17:22:13] <elukey>	 I see
[17:22:17] <Amir1>	 OMG the betafeatrures count take 7 hours
[17:26:38] <elukey>	 Amir1: one thing that I'd concentrate on is, if possible, to get a more verbose/specific error from the PHP code
[17:26:47] <elukey>	 because it seems a very generic "I failed"
[17:29:22] <elukey>	 also Amir1, can you try to sudo -u analytics-wmde cat /etc/mysql/conf.d/research-wmde-client.cnf ?
[17:29:25] <Amir1>	 The code doesn't have that :( https://github.com/wikimedia/analytics-wmde-scripts/blob/master/src/betafeatures/counts.php#L55
[17:29:47] <Amir1>	 elukey: yup, I'm afraid of sudo though :D
[17:30:06] <elukey>	 nono please do it
[17:30:13] <elukey>	 you should be able to run it
[17:30:13] <Amir1>	 let me try running sql queries with that
[17:30:23] <elukey>	 try to cat the file first :d
[17:30:55] <Amir1>	 did, got the output. the user is research and the password is m5MvEZwQeB7iVdU7UcCSc1p707
[17:31:09] <Amir1>	 elukey: ^
[17:31:19] <Amir1>	 Don't panic, I didn't put the actual passowrd
[17:31:24] <Amir1>	 it's random gen
[17:32:15] <elukey>	 :P
[17:32:26] <elukey>	 all right then let's see if the mysql command fails or not
[17:33:06] <elukey>	 but I think that it will not, the user/pass should work, people have been using them (I think) and I haven't heard a complain so far
[17:36:45] <Amir1>	 yeah it works fine
[17:36:47] <Amir1>	 weird
[17:39:23] <elukey>	 ok so we can probably rule out credentials
[17:40:38] <elukey>	 where is the PDO class defined? Maybe in there there is a way to add logging?
[17:40:45] <elukey>	 even manually to see what is happening
[17:41:03] <Amir1>	 yeah, on it
[17:41:28] <Amir1>	 https://github.com/wikimedia/analytics-wmde-scripts/blob/master/lib/WikimediaDb.php
[17:41:34] <Amir1>	 It's php's PDO
[17:42:17] <elukey>	 ah sorry super ignorant about it :)
[17:45:08] <Amir1>	 it returns only "False" there's nothing I can get out of it
[17:47:38] <elukey>	 Amir1: something like http://php.net/manual/en/pdo.errorcode.php ?
[17:48:39] <Amir1>	 Nice
[17:48:42] <Amir1>	 let me try that
[17:48:48] <Amir1>	 Probably this: http://php.net/manual/en/pdo.errorinfo.php
[17:49:32] <elukey>	 yeah let's try it
[17:58:26] <Amir1>	 https://grafana.wikimedia.org/d/000000270/echo-mention-status-notifications
[17:58:50] <Amir1>	 This one actually works, that's really interesting, if it was credential, this would also fail
[18:00:50] <Amir1>	 okay, I will check the result once it's done (in seven hours)
[18:07:49] <elukey>	 Amir1: all right, lemme know!
[18:08:01] <elukey>	 going off people, have a nice weekend! :)
[18:08:19] <Amir1>	 you too!
[18:44:55] <icinga-wm>	 PROBLEM - Check if the Hadoop HDFS Fuse mountpoint is readable on notebook1003 is CRITICAL: connect to address 10.64.21.109 port 5666: Connection refused
[18:51:31] <robh>	 heyas analytics folks, notebook1003 has an unusually high load and caused its icinga plugins to lag and false positive
[18:51:42] <robh>	 its ping otto but he isnt on irc =]
[18:51:45] <robh>	 id even
[18:52:18] <robh>	 its started to report back to icinga again, but the load spike could be indicative of a larger issue
[18:52:23] <robh>	 so i rather not ignore entirely ;]
[19:45:21] <icinga-wm>	 RECOVERY - Check if the Hadoop HDFS Fuse mountpoint is readable on notebook1003 is OK: OK
[21:15:22] <wikibugs>	 10Analytics, 10Research, 10Article-Recommendation: Generate article recommendations in Hadoop for use in production - https://phabricator.wikimedia.org/T210844 (10bmansurov)
[21:30:09] <wikibugs>	 (03PS1) 10Bmansurov: WIP: Oozie: Add article recomender job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/493762 (https://phabricator.wikimedia.org/T210844)
[21:30:35] <nuria>	 robh: k, taht machine is used by analysts to look at data it failing is not good but it is something we can look at Monday
[21:30:53] <robh>	 cool, then no need to worry, check!
[21:30:57] <robh>	 i admin logged the occurance
[21:34:03] <wikibugs>	 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10RyanSteinberg) Hi @bmansurov. I reviewed data today, specifically looking at section_id and freely_accessible elements.   section_i...