[00:26:16] 10Analytics-Kanban, 10Operations, 10Wikimedia-Stream, 10hardware-requests, 10Patch-For-Review: Decommission RCStream (rcs100[12]) - https://phabricator.wikimedia.org/T170157#3420970 (10Reedy) >>! In T170157#3443768, @Ottomata wrote: > Anyway, these hosts are off and ready for decom. Thanks! Umm.... ``... [00:32:43] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Improve purging for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933#3450897 (10Nuria) [00:35:36] 10Analytics-Kanban, 10Operations, 10Wikimedia-Stream, 10hardware-requests: Decommission RCStream (rcs100[12]) - https://phabricator.wikimedia.org/T170157#3450902 (10Reedy) [00:42:30] 10Analytics-Kanban, 10Operations, 10Wikimedia-Stream, 10hardware-requests: Decommission RCStream (rcs100[12]) - https://phabricator.wikimedia.org/T170157#3450931 (10RobH) [00:43:37] 10Analytics-Kanban: Mediawiki History Druid indexing failed - https://phabricator.wikimedia.org/T170493#3450934 (10Nuria) Actually, i take it back i think it should work even if inserting a subset of fields, maybe some fields have changed type? Rerunning to see [00:47:22] 10Analytics-Kanban, 10Operations, 10Wikimedia-Stream, 10hardware-requests: decommmission RCStream (rcs100[12]) - https://phabricator.wikimedia.org/T170157#3450935 (10RobH) [00:47:34] 10Analytics-Kanban, 10Operations, 10Wikimedia-Stream, 10hardware-requests: decommmission rcs100[12] - https://phabricator.wikimedia.org/T170157#3420970 (10RobH) [00:52:18] 10Analytics, 10Patch-For-Review, 10Performance: Add MediaWikiInstallPingback to EventLogging purging white-list - https://phabricator.wikimedia.org/T170986#3449888 (10Legoktm) What does the whitelist control? The schema should be receiving events already. [00:54:40] 10Analytics-Kanban, 10Operations, 10Wikimedia-Stream, 10hardware-requests, 10Patch-For-Review: decommission rcs100[12] - https://phabricator.wikimedia.org/T170157#3450991 (10Reedy) [00:59:23] 10Analytics-Kanban, 10Operations, 10Wikimedia-Stream, 10hardware-requests, 10Patch-For-Review: decommission rcs100[12] - https://phabricator.wikimedia.org/T170157#3451006 (10RobH) a:05RobH>03Cmjohnson [01:17:02] 10Analytics, 10Patch-For-Review, 10Performance: Add MediaWikiInstallPingback to EventLogging purging white-list - https://phabricator.wikimedia.org/T170986#3449888 (10Nuria) @Legoktm whitelist controls how data is purged after 90 days [01:17:47] 10Analytics-Kanban: Mediawiki History Druid indexing failed - https://phabricator.wikimedia.org/T170493#3451084 (10Nuria) Reruned and it failed: ------------------------------------------------------------------------------------------------------------------------------------ 0038299-170621131133576-oozie-oozi... [03:54:59] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3451260 (10Tbayer) For the record, @ellery and @MoushiraEl / @Moushira sadly no longer work for WMF, although it's possible that they still require ac... [04:05:35] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3451265 (10ZhouZ) I can confirm zhousquared belongs to me. [05:57:21] 10Analytics, 10Performance-Team: Add MediaWikiInstallPingback to EventLogging purging white-list - https://phabricator.wikimedia.org/T170986#3451342 (10Krinkle) 05Open>03declined This schema is and was never in use. The [MediaWikiInstallPingback](https://meta.wikimedia.org/wiki/Schema:MediaWikiInstallPingb... [08:08:37] 10Analytics-Kanban, 10DBA, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3448710 (10Marostegui) Could that be that db1047 was using utf8 just for historic reasons and dbstore1002 got the template from the dbstore servers that uses binary? [08:55:49] (03CR) 10Filippo Giunchedi: [V: 032 C: 032] Add scap3 config [analytics/statsv] - 10https://gerrit.wikimedia.org/r/363578 (https://phabricator.wikimedia.org/T129139) (owner: 10Filippo Giunchedi) [10:33:18] 10Analytics-Kanban, 10DBA, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3451799 (10elukey) >>! In T170952#3451463, @Marostegui wrote: > Could that be that db1047 was using utf8 just for historic reasons and dbstore1002 got the template from... [10:43:03] 10Analytics-Kanban, 10DBA, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3451817 (10Marostegui) Keep in mind that altering the database will not change the existing data, it will just change the default so the new tables will get the new def... [10:50:24] 10Analytics-Kanban, 10DBA, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3451829 (10elukey) >>! In T170952#3451817, @Marostegui wrote: > Keep in mind that altering the database will not change the existing data, it will just change the defau... [10:51:14] * elukey lunch! [10:52:17] 10Analytics-Kanban, 10DBA, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3451831 (10Marostegui) So what we can do is: ``` alter database log DEFAULT CHARACTER SET binary; ``` So the new tables will be binary at least, and if you feel some... [10:57:29] 10Analytics-Kanban, 10DBA, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3448710 (10jcrespo) A char with binary collation is essentially a binary field- mysql translates it automaticaly if you explicitly or implicitly define somethin as CHAR... [12:05:33] 10Analytics, 10Performance-Team: Add MediaWikiInstallPingback to EventLogging purging white-list - https://phabricator.wikimedia.org/T170986#3452083 (10mforns) Thanks for the prompt response! [12:24:18] 10Analytics, 10EventBus, 10Operations: Eventbus does not handle gracefully changes in DNS recursors - https://phabricator.wikimedia.org/T171048#3452124 (10elukey) [13:06:05] o/ [13:06:21] milimetric, around? [13:13:44] Looks like the large scale data analysis meeting is a bust today. [13:15:32] My agenda item was imagining a new flow of data that I'd like to show up in the data lake and other query-able sources. But this data flow has issues around free-text suppression. [13:21:36] essentially, I want to think about hooking ORES' new auditing system directly to EventBus and use that as a better way to get data to a few locations where it is needed. [13:21:50] Specifically batch querying and realtime random access. [13:24:52] 10Analytics, 10EventBus, 10Operations, 10User-Elukey: Eventbus does not handle gracefully changes in DNS recursors - https://phabricator.wikimedia.org/T171048#3452361 (10elukey) [13:38:33] PROBLEM - Hive Server on analytics1003 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hive.service.server.HiveServer2 [13:40:24] I am checking --^, I just merged a change in conf2001 to restrict zk rules, doesn't make sense [13:42:42] heap space issue, probably a huge query? [13:43:58] 10Analytics-Kanban, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3452419 (10Marostegui) This is the graph for the last 30 months of disk usage: https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=17&fullscreen&orgId=1&var-server=dbstore100... [13:44:33] RECOVERY - Hive Server on analytics1003 is OK: PROCS OK: 1 process with command name java, args org.apache.hive.service.server.HiveServer2 [13:46:02] restarted, will open a phab task later on [13:46:22] elukey: ok [13:46:31] it was because of a huge query? [13:46:44] ottomata: didn't check veerry carefully since I am merging a change for zk ferm rules :D [13:46:47] aye ok [13:46:50] i'll poke around a bit [13:46:51] thanks [13:47:10] I can clearly see a OOM for the hive server though [13:50:27] ottomata: now that I think about it I didn't change Xmx settings on an1003 [13:50:36] err Xms [13:50:55] and we don't have it in the analytics-hadoop board [13:51:54] aye, but it wasn't a hadoop worker to be audited [13:51:56] which makes sense [13:52:13] select count(*) from wmf.webrequest where year=2017 and uri_host like '%wikiknihy.cz% [13:52:14] ? [13:52:38] it seems that one, Filippo might have run it [13:52:56] hmmm, it seems maybe that one finisehd though Completed executing command(queryId=hive_20170719130808_b8055293-50c2-4ad8-acc2-bd67c5b8877c); Time taken: 1608.244 seconds [13:53:31] maybe the results were just too big, and it couldnt' send them back [13:53:31] hm [13:54:56] ottomata: (unrelated) FYI - https://gerrit.wikimedia.org/r/#/c/366228/2/hieradata/role/common/configcluster.yaml [13:55:19] I created a new profile this morning and now I am running puppet [13:55:28] Moritz is checking if any request gets dropped [13:55:59] coool! [13:56:07] conflighcluster proflie, huh [13:56:12] oook [13:56:41] nothing got dropped so far, I think we're all good [13:56:55] elukey: also unrelated, i need to add those indexes to the slaves [13:57:02] the el slaves [13:57:06] is it safe to do so now? [13:57:34] the script is still running so it might be better to wait a bit, but if you are really in a hurry I gues we can try [14:01:03] 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Ensure indexes are added to `meta_dt` and unique `meta_id` fields in eventbus MySQL tables in eventlogging databases. - https://phabricator.wikimedia.org/T170925#3452487 (10Ottomata) @kaldari: https://wikitech.wikimedia.org/wiki/EventBus#Metad... [14:01:52] ok, no not in a hurry at all [14:01:55] you let me know ya? [14:02:21] ottomata: sure! [14:06:00] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10DBA, and 2 others: Add index to mediawiki_page_create_1 table - https://phabricator.wikimedia.org/T170990#3452501 (10Ottomata) We can add extra indexes to MySQL tables, but they'll be one offs. That is, other EventBus event tables that we m... [14:16:05] 10Analytics, 10Wikimedia-Stream, 10Services (watching): Update EventStreams to service-template-node v0.5.2 - https://phabricator.wikimedia.org/T171011#3452536 (10Ottomata) [14:18:01] 10Analytics-Kanban, 10Operations, 10Wikimedia-Stream, 10hardware-requests, 10Patch-For-Review: decommission rcs100[12] - https://phabricator.wikimedia.org/T170157#3452538 (10Ottomata) > Because of in CommonSettings.. Aye yai yai. @Reedy very sorry about that, thanks for cleaning up my mess. [14:20:14] elukey: can we do ops sync tomorrow? [14:20:53] ottomata: sure [14:32:07] zookeeper ferm rules updated! \o/ [14:32:13] and nothing exploded [14:32:28] now coffee + check what happened to an1003 [14:42:29] great! [14:42:36] elukey: i'm looking into the druid mw history thing [14:42:40] found the exception on the druid middlemanager [14:42:44] but it isn't very informative [14:42:55] i don't know if druid submits a yarn job to do the actual loading of data [14:43:00] or if it just loads from hdfs [14:43:04] it looks like it submits a yarn job [14:43:09] but i can't find the app id [14:43:13] that i bet has more interesting logs [14:44:08] ahhh interesting! [14:54:40] 10Analytics-Kanban, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3452667 (10Marostegui) All these wikis are not InnoDB compressed: ``` 66G viwiki 97G ukwiki 32G kowiki 82G huwiki 43G hewiki 48G frwiktionary 69G fawiki 131G eswiki 52G cawiki 145G arwiki ``` So I will c... [15:00:41] a-team: holaaaa [15:00:50] hellooo [15:01:07] ah, double factor auth now... [15:08:11] sorry halfak, was at the doctor's this morning. I have to go weekly now, so I'll probably miss all our Wednesday time slots [15:08:35] let's chat about the new data though, sounds interesting [15:22:53] 10Analytics-Cluster, 10Analytics-Kanban, 10Security, 10User-Addshore: Access rights for HDFS on stat100* for Sqoop tasks - https://phabricator.wikimedia.org/T170052#3452754 (10Milimetric) it's good that it worked, Goran. Stat1005 is not yet fully vetted and ready to go, so we don't expect everything to wo... [15:54:28] No worries milimetric, we should reschedule. [15:55:18] yeah, I'm free afternoons usually, would that work before joseph's back halfak? [15:55:38] milimetric, I think that would be hard for Goran if he plans to keep attending. [15:56:06] Maybe we could move an hour later. E.g. 10AM EDT [15:56:15] On... Fridays? [15:56:26] yeah, Fridays would be fine anytime [15:56:36] just Wednesdays I have the doc checkup [15:56:44] Looks like you have ops duty in the beginning of Aug [15:56:50] at that time [15:56:58] oh, don't worry about that, baby will def. come by then :) [15:57:01] Arg! During wikimania! [15:57:05] lol [15:57:06] kk [15:57:32] so yeah, this is short term until j's back, you two can figure it out after that [15:57:48] gonna go take lunch now, see you Friday [15:58:09] cool [15:58:09] o/ [16:05:18] ottomata: regarding the druid indexing problems [16:06:56] ottomata: it is real hard to find the application id but even then i cannot find that log with: sudo -u hdfs hdfs dfs -ls /var/log/hadoop-yarn/apps/hdfs/logs | grep 1498042433999_104022 [16:11:04] 10Analytics-Kanban, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3452958 (10Marostegui) >>! In T168303#3452429, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://tools.wmflabs.org/sal/log/AV1bGWR-wg13V6285OFz} [2017-07-19T13... [16:12:47] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3452964 (10Nuria) @kladari, how about you put your scripts in : git clone https://gerrit.wikimedia.org/r/analytics/report... [16:14:03] 10Analytics-Kanban: Mediawiki History Druid indexing failed - https://phabricator.wikimedia.org/T170493#3452970 (10Nuria) Looks like issues are related to having changed timestamp field: https://github.com/wikimedia/analytics-refinery-source/commit/7a887f2f31abdf2121d5d3ecfd0fb0e0e528bd53 [16:14:04] nuria_: yeah it was super hard for me to find it [16:14:18] i looked in job browser in hue and searched for jobs submitted by druid [16:14:22] but [16:14:33] it ran as druid, not hdfs [16:14:33] so [16:14:51] ottomata: ooohhhh now i get it [16:15:09] sudo -u druid yarn logs -applicationId application_1498042433999_104022 [16:15:48] ottomata: super thnaks [16:16:32] ottomata: will try to fix ingestion today [16:44:49] all right going offline people! [16:44:51] * elukey off [16:54:34] 10Analytics-Cluster, 10Analytics-Kanban, 10Security, 10User-Addshore: Access rights for HDFS on stat100* for Sqoop tasks - https://phabricator.wikimedia.org/T170052#3453115 (10GoranSMilovanovic) @Milimetric It worked thanks to the effort that you've put. Thank you so much. But at some point, we should try... [16:54:45] 10Analytics-Cluster, 10Analytics-Kanban, 10Security, 10User-Addshore: Access rights for HDFS on stat100* for Sqoop tasks - https://phabricator.wikimedia.org/T170052#3453117 (10GoranSMilovanovic) 05Open>03Resolved a:03GoranSMilovanovic [16:58:46] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3453133 (10kaldari) @Nuria: That link doesn't work for me. Are you suggesting I create that repo? Just want to make sure i... [17:03:15] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3453141 (10Nuria) @kaldari repo exists in gerrit, you can just clone it. [17:16:20] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10DBA, and 2 others: Add index to mediawiki_page_create_1 table - https://phabricator.wikimedia.org/T170990#3453181 (10kaldari) >We can add extra indexes to MySQL tables, but they'll be one offs @Ottomata: All of the existing EventLogging tabl... [17:18:03] 10Analytics-Cluster, 10Analytics-Kanban: Install Java 7 in Debian Stretch (to fix sqoop (and possibly other things) on stat1005) - https://phabricator.wikimedia.org/T171072#3453190 (10Ottomata) [17:21:27] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3453225 (10kaldari) @Nuria: Oops, you're right. My bad. Didn't notice the "git clone" in front of it :) [17:21:45] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10DBA, and 2 others: Add index to mediawiki_page_create_1 table - https://phabricator.wikimedia.org/T170990#3453226 (10Ottomata) > All of the existing EventLogging tables (not from EventBus) have wiki, timestamp 2-column indexes ```KEY `wiki... [17:55:29] milimetric: yt? [17:55:57] milimetric: i think i am missing something big time, should this query not return "some" results? "select * from mediawiki_history where snapshot=2017-06 limit 5;" [17:56:47] 10Analytics-Kanban: Mediawiki History Druid indexing failed - https://phabricator.wikimedia.org/T170493#3453375 (10Nuria) Note to self that druid jobs run as druid not hdfs and thus logs bneed to be fetched like: sudo -u druid yarn logs -applicationId application_blah [17:56:58] cc mforns [17:57:25] nuria_, yes, looks like [17:58:21] nuria_, maybe the snapshot needs a day? [17:58:28] mforns: it should return data right? [17:58:32] not sure though [17:58:41] did you see which snapshots are available? [17:58:41] mforns: [17:58:45] https://www.irccloud.com/pastebin/YsZe9yHD/ [17:59:04] mforns: ok, have a meeting , will continue in a sec [17:59:07] k [18:00:07] oh, I think the quotes are missing in the snapshot, I guess it's trying an integer subtraction instead [18:01:11] mforns: oohhh, [18:01:47] hehehe [18:04:00] sorry, missed this ping [18:04:08] yes, quotes [18:04:20] I did that too, no worries [18:09:34] elukey: still on? [18:15:14] 10Analytics-Cluster, 10Analytics-Kanban: Install Java 7 in Debian Stretch (to fix sqoop (and possibly other things) on stat1005) - https://phabricator.wikimedia.org/T171072#3453492 (10Ottomata) Yar, I think this is not going to work. I tried on a Stretch labs instance. There a few (innocuous) dependencies th... [18:15:39] 10Analytics-Cluster, 10Analytics-Kanban, 10Security, 10User-Addshore: Access rights for HDFS on stat100* for Sqoop tasks - https://phabricator.wikimedia.org/T170052#3418073 (10Ottomata) Ooof stat1005 nastiness. Thanks for the heads up. I filed T171072. [18:16:13] 10Analytics-Kanban, 10Documentation, 10Services (watching): Document revision-create event for EventStreams - https://phabricator.wikimedia.org/T169245#3453506 (10Ottomata) I think we should wait til we get revision-score, and then announce. [18:35:44] (03PS6) 10Ottomata: Add oozie util workflow to launch spark jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/201009 (https://phabricator.wikimedia.org/T94596) [18:36:15] (03CR) 10Ottomata: "Done, but instead of --num-executors, I used your conf from mw history to set spark_max_executors, to use dynamic allocation by default." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/201009 (https://phabricator.wikimedia.org/T94596) (owner: 10Ottomata) [18:43:46] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3453684 (10hoo) I would like to keep my access (along with my deploy shell access), as I'm using it from time to time. See also @daniel's comment above. [19:37:32] 10Analytics, 10Community-Liaisons, 10User-Johan: Collect information about how we collect user statistics in one place - https://phabricator.wikimedia.org/T132405#3453904 (10Johan) 05Open>03declined Let's be realistic: I want to do this but will //never// find the time. [19:41:43] 10Analytics-Dashiki, 10Analytics-Kanban, 10MW-1.30-release-notes (WMF-deploy-2017-06-13_(1.30.0-wmf.5)), 10Patch-For-Review, 10Wikimedia-log-errors: Warning: JsonConfig: Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found - https://phabricator.wikimedia.org/T166335#3453939 (10... [19:51:32] (03PS7) 10Ottomata: Add oozie util workflow to launch spark jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/201009 (https://phabricator.wikimedia.org/T94596) [19:54:59] 10Analytics, 10Community-Liaisons, 10User-Johan: Collect information about how we collect user statistics in one place - https://phabricator.wikimedia.org/T132405#3454000 (10Johan) 05declined>03Open a:05Johan>03None [20:05:47] 10Analytics, 10Analytics-Cluster: CamusPartitionChecker does not work when topic names have '.' or '-' in them. - https://phabricator.wikimedia.org/T171099#3454060 (10Ottomata) [20:05:55] 10Analytics, 10Analytics-Cluster: CamusPartitionChecker does not work when topic names have '.' or '-' in them. - https://phabricator.wikimedia.org/T171099#3454072 (10Ottomata) p:05Triage>03Low [20:17:59] (03PS1) 10Nuria: [WIP] Modifying ingestion spec after additions to edit history [analytics/refinery] - 10https://gerrit.wikimedia.org/r/366327 (https://phabricator.wikimedia.org/T170493) [20:19:48] nuria_: hiIiIIIIi got a few mins for a brain bounce about eventlogging refine stuff? [20:20:03] ottomata: i have a meeting in 10 mins , so yes [20:20:07] k [20:20:17] pfct bcave [20:20:48] omw [20:33:12] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Contributors-Analysis, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3454258 (10kaldari) Here's another example of the query script set up: https://gerrit.wikimedia.org/r/#/c/322007/ [21:02:57] 10Analytics-Kanban, 10WMF-Legal, 10Services (watching): License for pageview data - https://phabricator.wikimedia.org/T170602#3454327 (10mforns) Thanks @GWicke! One question though: I think that the license paragraph in https://wikimedia.org/api/rest_v1/ is not specified in [[ https://github.com/wikimedia/r... [21:03:27] 10Analytics-Dashiki, 10Analytics-Kanban, 10MW-1.30-release-notes (WMF-deploy-2017-06-13_(1.30.0-wmf.5)), 10Patch-For-Review, 10Wikimedia-log-errors: Warning: JsonConfig: Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found - https://phabricator.wikimedia.org/T166335#3454330 (10... [21:08:15] 10Analytics-Dashiki, 10Analytics-Kanban, 10MW-1.30-release-notes (WMF-deploy-2017-06-13_(1.30.0-wmf.5)), 10Patch-For-Review, 10Wikimedia-log-errors: Warning: JsonConfig: Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found - https://phabricator.wikimedia.org/T166335#3454359 (10... [21:11:04] 10Analytics-Dashiki, 10Analytics-Kanban, 10MW-1.30-release-notes (WMF-deploy-2017-06-13_(1.30.0-wmf.5)), 10Patch-For-Review, 10Wikimedia-log-errors: Warning: JsonConfig: Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found - https://phabricator.wikimedia.org/T166335#3454369 (10... [21:13:51] RainbowSprinkles: we can chat here if you want - do you know anyone that can help debug the problem? [21:14:14] Well, I'm not a huge fan of JsonConfig to begin with [21:14:32] agreed [21:14:37] It's been spewing warnings and errors in production since the day it was deployed [21:14:46] And the person who wrote it (and understands it) is no longer with us [21:14:51] That is problematic [21:15:00] I thought that was fairly normal btw, since I fixed a null pointer exception in Kunal's code that was there for like a year [21:15:25] What's normal? Having errors in production? [21:15:32] yeah, my long term thought on JsonConfig is to migrate everything using that to structured data (but maybe I don't understand the aims of that project) [21:15:40] Yes. But it's good to have someone who can find and fix an issue. /Completely unsupported/ code is a liability [21:15:52] yeah, I thought errors in production were super normal since I acted all like "WTF" when I saw that null pointer and people shrugged [21:16:06] (A NPE in production for the year should be the exception, not the norm) [21:16:16] nono, we support this code, it's just a weird error that 1. doesn't seem to affect anything and 2. can't be replicated as far as we can tell [21:16:26] gotcha, adjusting expectations [21:16:38] (I take production errors Very Srsly) [21:16:57] Noisy errors: makes it impossible for me to have insight into our deploys :) [21:17:02] Anyway, I digress. [21:17:12] Ok, I understand you support it. But clearly it's not easy to replicate. [21:17:15] Sometimes we gotta debug in prod [21:17:18] Add extra logging [21:17:19] Etc [21:17:43] right, I think the extra logging will go into JsonConfig, I'll submit a patch to that then [21:17:49] RainbowSprinkles: remind me -- is JsonConfig that thing that uses RobLa's json schema library? [21:17:52] shall I self-merge? [21:17:57] harej: no [21:17:57] harej: Nope [21:18:00] oh, okay [21:18:06] yuri's thing [21:18:06] * harej returns to obscurity [21:18:11] milimetric: I think that's find [21:18:15] *fine [21:18:27] k, I'll do that next, hopefully before baby comes!! :) [21:18:40] hey y'all - yarn.wikimedia is probably currently down because of some ldap CA related issues - you might have to run puppet and restart apache to bring it back up. [21:18:55] ooh, ottomata ^ [21:18:59] ooh, he's gone [21:19:09] milimetric: ah okay - may be i can [21:19:11] madhuvishy: none of us have rights, but that's just for monitoring [21:19:29] wouldn't worry too much... though it'll make debugging harder till 'morrow [21:22:55] it's fine. fixed [21:24:20] hue, yarn, and pivot I think :) [21:32:02] thanks, madhu [21:32:17] oh yeah, pivot would be pretty important [21:38:32] 10Analytics-Kanban, 10WMF-Legal, 10Services (watching): License for pageview data - https://phabricator.wikimedia.org/T170602#3454471 (10GWicke) >>! In T170602#3454327, @mforns wrote: > Thanks @GWicke! > > One question though: I think that the license paragraph in https://wikimedia.org/api/rest_v1/ is not s... [21:47:22] 10Analytics-Dashiki, 10Analytics-Kanban, 10MW-1.30-release-notes (WMF-deploy-2017-06-13_(1.30.0-wmf.5)), 10Patch-For-Review, 10Wikimedia-log-errors: Warning: JsonConfig: Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found - https://phabricator.wikimedia.org/T166335#3292968 (10... [21:50:06] 10Analytics-Dashiki, 10Analytics-Kanban, 10MW-1.30-release-notes (WMF-deploy-2017-06-13_(1.30.0-wmf.5)), 10Patch-For-Review, 10Wikimedia-log-errors: Warning: JsonConfig: Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found - https://phabricator.wikimedia.org/T166335#3454504 (10... [22:45:17] hola! did anything change with respect to mysql on stat1006 ? i can't seem to query the EventLogging database any more... [22:45:54] not sure if it's related to https://phabricator.wikimedia.org/T152712 (i was previously using stat1003) [23:12:38] jdlrobson: 1006 is not ready for use yet, you might to try 1002 [23:21:00] nuria_: the issue is with the mysql db i cant seem to connect any more [23:21:09] not sure if my rights got revoked or password changed..