[08:54:05] <wikibugs>	 10Analytics, 10Analytics-Kanban: JVM pauses cause Yarn master to failover - https://phabricator.wikimedia.org/T206943 (10elukey) p:05Triage>03High
[08:54:08] <elukey>	 joal: --^ :(
[08:56:00] <joal>	 High elukey  - This is not nice
[08:57:43] <elukey>	 joal: I just restarted the failed job
[08:57:58] <elukey>	 missed to log it since nobody was online :)
[08:58:06] <joal>	 np elukey - many thanks for that
[08:58:31] <joal>	 elukey: I'd be interested in investigating more on this issue with you if you'd like
[08:58:33] <elukey>	 so I basically modified spark 2.3.0 with spark 2.3.1
[08:58:44] <elukey>	 fixed manually and restarted the coordinator
[08:58:48] <joal>	 elukey: I'm not so good in JVM tuning, and learning with you is great
[08:59:08] <elukey>	 it is weird since I don't see stressful signs in the GC logs
[08:59:11] <elukey>	 sigh
[08:59:15] <joal>	 meh :(
[08:59:26] <elukey>	 and the fact that it happens right after changing the masters is not good
[08:59:43] <joal>	 elukey: the spark default was not correct in properties file?
[09:00:02] <joal>	 agreed elukey - Could be material issue (RAM or something?)
[09:00:11] <elukey>	 was 2.3.0, I think that Andrew cleaned up some oozie dirs on hdfs since I can find only "spark" and spark2.3.1
[09:00:21] <elukey>	 (among the oozie libs)
[09:00:35] <elukey>	 how are you feeling btw?
[09:01:16] <joal>	 elukey: better!! Still some leftover, but I can now understand reasonably what I'm looking at :)
[09:02:46] <joal>	 elukey: I'm very sorry, the mobile_apps_session_metrics spark-lib param shold have been updated in my previous patch, but I missed it
[09:02:50] <joal>	 changing it now
[09:03:33] <elukey>	 joal: I hope that I've restarted the coordinator with the right timing
[09:03:40] <elukey>	 this one is 7 days and it was a bit weird
[09:04:15] <wikibugs>	 (03PS1) 10Joal: Correct spark-lib version in apps-session-metrics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/467112
[09:04:33] <joal>	 elukey: --^
[09:04:38] <joal>	 elukey: will self merge that now
[09:05:11] <wikibugs>	 (03CR) 10Elukey: [C: 032] Correct spark-lib version in apps-session-metrics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/467112 (owner: 10Joal)
[09:05:33] <wikibugs>	 (03CR) 10Joal: [V: 032 C: 032] "Self merging bug-correction" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/467112 (owner: 10Joal)
[09:08:00] <joal>	 elukey: From what I can see, the job you restarted has correct params :)
[09:08:06] <joal>	 Many thanks again
[09:09:37] <elukey>	 \o/
[09:15:00] <elukey>	 !log restart apps-session-metrics with spark 2.3.1 oozie libs (modified the coordinator.properties file manually on disk)
[09:15:01] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[09:15:24] <elukey>	 !log restart yarn resource manager on an-coord1002 (failover happened due to jvm issues)
[09:15:25] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[09:15:54] <joal>	 Oh by the way elukey - You could have restarted with "-Doozie_spark_lib=spark2.3.1" instead of modifying file on disk ;)
[09:17:11] <elukey>	 ah snap you are right!
[09:17:15] <elukey>	 way easier sigh
[09:17:27] <joal>	 elukey: no big deal - justb saying
[09:21:04] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Russian-Sites: Wikistats summary for Report Card has English/Russian Wikipedia-Zero traffic as separate entries instead of added to English/Russian Wikipedia mobile traffic - https://phabricator.wikimedia.org/T146777 (10Liuxinyu970226)
[09:26:38] <elukey>	 joal: have a nice sunday! Talk with you tomorrow :)
[09:26:46] <joal>	 Bye elukey :)
[15:09:11] <wikibugs>	 10Quarry: Provide a way to add hyperlink in Quarry results/output - https://phabricator.wikimedia.org/T74874 (10Seb35) I like Tgr’s both proposals, even if the implementation is nontrivial for the first comment.  Perhaps for the format page_title it can be used the MediaWiki API with cached results for e.g. 7 da...
[15:16:34] <wikibugs>	 10Quarry: Provide a way to add hyperlink in Quarry results/output - https://phabricator.wikimedia.org/T74874 (10Seb35) A more advanced suggestion (but could be another task) would be to have a dictionary about the datatype of common field names. E.g. default formatting for `rev_user_text` is `@quarry:format link...
[15:53:57] <wikibugs>	 10Analytics, 10EventBus, 10Core Platform Team Current (Doing), 10Patch-For-Review, 10Services (doing): Revision visibility change event sets a wrong performer - https://phabricator.wikimedia.org/T206277 (10CCicalese_WMF)
[16:28:16] <icinga-wm>	 PROBLEM - Webrequests Varnishkafka log producer on cp5007 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf
[16:28:17] <icinga-wm>	 PROBLEM - eventlogging Varnishkafka log producer on cp5007 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf
[16:28:18] <icinga-wm>	 PROBLEM - statsv Varnishkafka log producer on cp5007 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf
[16:33:37] <icinga-wm>	 RECOVERY - Webrequests Varnishkafka log producer on cp5007 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf
[16:33:47] <icinga-wm>	 RECOVERY - eventlogging Varnishkafka log producer on cp5007 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf
[16:33:47] <icinga-wm>	 RECOVERY - statsv Varnishkafka log producer on cp5007 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf
[17:14:49] <wikibugs>	 10Quarry: Provide a way to add hyperlink in Quarry results/output - https://phabricator.wikimedia.org/T74874 (10Seb35) I would have a first fix in JavaScript for the URLs, which is a simpler case, but I’m not sure it is really useful given there are not so much URLs in MediaWiki databases. Probably we should ins...
[19:42:07] <wikibugs>	 10Analytics, 10Analytics-EventLogging: It should be possible to see the number of server-side validation errors for a schema in Grafana - https://phabricator.wikimedia.org/T206959 (10phuedx)
[21:57:07] <tgr>	 I just got an oozie job error report (mediawiki-load-wf-wmf_raw.ApiAction-2018-10-14-19) that I don't understand (probably not the report's fault, I just don't know anything about oozie)
[21:57:16] <tgr>	 am I supposed to do something about it?
[22:07:11] <wikibugs>	 10Quarry: Provide a way to add hyperlink in Quarry results/output - https://phabricator.wikimedia.org/T74874 (10Tgr) >>! In T74874#4664521, @Seb35 wrote: > I would have a first fix in JavaScript for the URLs, which is a simpler case, but I’m not sure it is really useful given there are not so much URLs in MediaW...