[00:46:38] bd808: Can https://github.com/wikimedia/operations-mediawiki-config/commit/d73aaff32b be reverted now that utf-8 is fixed? [00:46:47] Or do we want to disable zero in logstash for others reasons. [00:46:56] I imagine other channels can equally produce such urls or values in log messages [00:47:11] utf-8 isn't fixed [00:48:33] we fixed it so URLs don't blow things up when monolog adds them (url and referrer) but any message that has non-utf-8 chars will not make it to logstash [00:48:43] this has actually always been the case [00:48:57] but not monolog raises an exception when it happens [00:49:02] which is not goodly [00:49:17] zero was triggering this regularly [00:51:05] I'm probably going to have to find a way to convince upstream that what they did there (raising exceptions on json encoding failure) is a really bad idea [00:51:33] much better to quitely drop the log event or at most raise a warning [00:53:57] Krinkle: were you really wanting zero logs for something or just generally wanting to see us not exclude things for dumb reasons? [00:54:58] bd808: Right. We don't want run time exceptions for logging failures. [00:55:07] Same way we don't want the site to be down if logstash is unavailable [00:55:13] agreed [00:55:40] bd808: I heard some talk about using local app server's syslog and aggregating from there instead of sending over udp directly [00:55:58] Is that still the plan / already happened / decided not to/ later? [00:56:25] we could do it if there was a good reason to [00:56:44] bd808: I thought it was a post-mortem action item when the site when down because of logstash downage? [00:57:04] we were logging via redis then [00:57:13] now we are doing syslog udp datagrams [00:57:17] bd808: as for zero, mostly the latter. If we're excluding it because logstash throws for non-utf8, disabling zero is not an adequate fix. [00:57:23] monolog* [00:57:50] bd808: syslog and udp seem mutually exclusive. What does that mean [00:57:58] there are 54 "JSON encoding failed" runtime errors in the current log file [00:58:04] We emit to central server directly, but using a syslog-like format over udp? [00:58:10] (which doesn't care that it's received I suppose) [00:58:24] the syslog format. syslog is a transport protocol [00:58:30] Right [00:59:21] so today we create udp datagrams with a payload of a syslog encoded message and fling them towards logstash [00:59:27] which is all we needed [01:00:06] the tradeoff is that now we don't have guaranteed delivery for log events [01:00:22] but we haven't ever had that with udp2log either so not a huge deal [01:00:35] Yeah [01:00:49] bd808: So we sent to logstash udp and fluorine over udp from app servers directly [01:00:57] to two places [01:00:57] yeah [01:01:06] one does not consume from the other [01:01:06] ok [01:01:22] different payloads (json vs lines) [01:01:39] right [01:02:11] and different channels. we don't send api.log or most runJobs.log and some other spammy things to logstash [01:02:39] the runJobs thing was new last night and cut out ~66% of the logstash event load [01:03:08] but this encoding things needs to get fixed. the two hack we put in are not getting all of it [01:03:17] I'll open a bug [01:03:24] and probably a hot patch [01:10:59] Krinkle: https://phabricator.wikimedia.org/T118057 [01:12:06] bd808: For runJobs, it seems like some of that should just not exist to begin with? We don't stash debug level for anything outside testwikis, right? [01:12:14] Or do we only ignore it for non-monolog [01:12:28] it seems logger->debug() is tracked in logstash [01:12:34] we only ignore it for logstash [01:12:40] Right [01:13:03] hm.. so the massive load from runjobs is a different level [01:13:22] I'd be nice to keep error level potentially [01:13:24] if it's not too much [01:13:27] from runJobs [01:13:36] And/or adjust those messages to be in a more appropiate level [01:14:14] Getting it from fluorine works for now as well, but it'd benice that if we see error levels rising for jobs in grafana, that we can go into logstash and get a few samples [01:15:27] we do still have warnings and errors [01:15:44] I ajusted the logging in runJobs to do that (full PSR3) [01:19:11] https://github.com/wikimedia/mediawiki/commit/2eea1d5a42a4b5c455b90a4ed61bf0f9c9296141 [01:19:13] Ah, that was you [01:20:58] And https://github.com/wikimedia/operations-mediawiki-config/commit/4ee33b68bb1c10c7f7cbf5299a3faaa291aff50a [01:21:00] okay, perfect :) [04:40:20] ori: by backward-incompatible, i just mean that on-wiki modules will have to do slightly more work than replace require('Module:Arguments') with require('getArgs') if they want to use it. it won't actually break anything on its own though [04:40:49] the main thing i wanted looked at was how i did arg translation [18:05:10] legoktm: arghhhhhhhhhhhhhhhh why is oojs-ui loading on every page againnnn [18:06:38] * MatmaRex brings the pitchfork [18:13:06] ori: it's VE and its 'ext.visualEditor.switching' module. [18:13:32] how did you figure that out so quickly? [18:14:17] it'd be nice to automate that if possible.. mw.loader.inspect.whyTheFuck( moduleName ) [18:14:54] caused by b550323b534b59622afefdf9bf9fbc23f177f46f [18:15:29] ori: i took the list from https://en.wikipedia.org/w/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector , got all the modules that depend on 'oojs-ui' (or 230, for me at least), and ran mw.loader.getState() in the console for each one [18:15:39] and that was the only 'ready' one [18:16:30] are you updating the task with this or should i? [18:16:43] no, please do. there's a task? [18:16:55] i just re-opened https://phabricator.wikimedia.org/T112401 [18:17:07] it's not Echo's fault, anyway [18:17:35] pretty sure it's https://gerrit.wikimedia.org/r/#/q/If03dab6130aed7662b04000b809884a514bb2762,n,z [18:18:21] Yeah, I think you're right. But in this case it's useful for the discussion to happen on the same task, because it's the same underlying issue. [18:18:31] ori: so, when are you going to work on splitting up the module? :) [18:18:35] Massive library, commonly used, easy to load without realizing [18:19:14] haven't you heard? I transitioned from being a mediocre programmer to being an even more mediocre manager [18:19:23] i just ask annoying questions now [18:19:47] truly the management disease is spreading [18:19:56] heheh [18:21:47] (eh, i'd put this in a separate task, that one was all about Echo) [18:22:43] too late, i already made the case for keeping it in that task [18:22:58] i would have made a new one if you were quicker to object :P [18:25:02] What happened to being an anti-performance engineer? [18:26:30] relegated to a hobby [18:40:02] MatmaRex: mw.loader.inspect.whyTheFuck( moduleName ) <-- looks like i did this already [18:40:06] mw.inspect.getDependencyGraph() [18:40:30] http://i.imgur.com/tmgSV1O.png [18:40:56] oh, hah [22:14:28] hello [22:14:32] anybody know anything about redis? [22:14:41] i've got cache working well but i need to enable the jobrunner [22:14:50] i found this: https://github.com/wikimedia/mediawiki-services-jobrunner but need a little guidance [22:18:35] You need to ask a more specific question really :P [22:19:20] trying to configure job queue as per: https://www.mediawiki.org/wiki/Redis#Job_queue [22:20:55] but i am missing something that is not specified in the instructions. when included in my localsettings, i get the error: Exception from line 92 of /var/www/html/w/includes/jobqueue/JobQueueRedis.php: Non-daemonized mode is no longer supported. Please install the mediawiki/services/jobrunner service and update $wgJobTypeConf as needed. [22:21:39] looking for help on how to install the mediawiki/services/jobrunner service [22:21:51] i found the git project, but no installation instructions [22:21:59] https://github.com/wikimedia/mediawiki-services-jobrunner [22:22:06] it is deployed on all wmf wikis afaik [22:22:33] https://github.com/wikimedia/operations-puppet/blob/812f280d16acfe3083259e8dfa7ce12ebf71da87/modules/mediawiki/files/jobrunner.conf [22:23:28] oic, install as system service [22:23:57] i'm on debian stretch, could you gimem a clue as to what and where i need to clone? [22:27:35] All the config will be in the puppet repo... But it's not always easy to follow if oyu're not so used to puppet [22:28:43] yeah i have no idea what puppet is [22:28:54] if you point me int he right direction i can work it out [22:29:38] https://github.com/wikimedia/operations-puppet is the repo (or the github sync of it) [22:31:35] got it, so it is a package manager of sorts [22:31:51] It's a lot more than that [22:32:53] configuration management [22:34:27] right [22:35:13] there are no install instructions on the article, are you able to help me set it up? [22:36:23] I don't know really how to suggest to set it up for a third party, as the config to host stuff is wmf specific [22:37:04] ok so i really only need redis job runner [22:37:19] i don't need to use puppet to manage its configuration if it can be run standalone [22:37:22] let me see [22:37:52] you could just run it in screen if you really wanted, but it probably doesn't help much on reboot etc :P [22:42:13] well i am running runJobs.php via cron every hour currently. instead, i could write an init script to have the process initialise at boot [22:42:27] i just need to figure out where to reasonably store the code [22:43:01] really an extension should be written [22:43:20] How would an extension help? [22:43:58] it would allow the service to be monitored and executed by the web server [22:44:22] if it is a simple script which runs in the background, then i suppose an extension is unnecessary [22:44:33] but some clearer instructions for end-users certainly are :) [22:44:47] i'm writing up the issues i've had on the Redis article on media-wiki.org [22:45:27] in case anyone else tries this (i expect they will, as it has been advised by the SMW team as a more responsive cache than memcached or SQL [22:45:28] ) [22:45:43] for new features they have released in 2.3 [22:53:57] https://www.mediawiki.org/wiki/Topic:Ss9ues5n7gtctppm [23:51:55] TIL there are web browsers in the wild that use non-UTF-8 characters in their User-Agent headers :/ [23:52:15] not surprising I suppose but annoying [23:52:55] And rediscovered that we have user names in encodings other that UTF-8 [23:53:09] to go with our article names that are also not in UFT-8