[00:19:12] 10Striker: Striker error log events not getting into ELK cluster due to UDP truncation of JSON payload - https://phabricator.wikimedia.org/T151422#2899986 (10bd808) Apparently californium can't negotiate a packet size larger than 1500 bytes with the logstash servers. ``` californium:~ bd808$ python3 Python 3.4.3... [00:38:38] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: Update Mono to newer stable release - https://phabricator.wikimedia.org/T152949#2900028 (10MaxBioHazard) But how I can do it without sudo? [00:39:40] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:24:41] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2026836 (10Liuxinyu970226) Per T136910 [01:39:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:05:40] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:15:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:04:04] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: Update Mono to newer stable release - https://phabricator.wikimedia.org/T152949#2864391 (10zhuyifei1999) http://www.mono-project.com/docs/compiling-mono/linux/ Use a prefix somewhere within your home dir, such as `~/.mono` [07:06:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:47:02] PROBLEM - Free space - all mounts on tools-worker-1003 is CRITICAL: CRITICAL: tools.tools-worker-1003.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found) tools.tools-worker-1003.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-worker-1003.diskspace.root.byte_percentfree (<40.00%) [10:11:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:37:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:42:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:38:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:27:34] can someone restart https://tools.wmflabs.org/tusc/ ? [14:58:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:33:03] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:05:33] 06Labs, 10Tool-Labs, 10video2commons: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2900654 (10zhuyifei1999) [17:34:30] 06Labs, 10Tool-Labs, 10video2commons: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2900947 (10zhuyifei1999) The same applies whether or not the file actually exists: eg non-existent file h... [18:16:05] 06Labs, 10Tool-Labs, 10video2commons: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2900962 (10zhuyifei1999) The exception seems to be from flask: (manually formatted) ``` Traceback (most r... [18:26:51] Steinsplitter: magnus decomissioned and stopped it I think. you've to ask him :) [18:29:08] 06Labs, 10Tool-Labs, 10video2commons: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2900964 (10zhuyifei1999) I added two lines before `raise BadRequest()`: ``` __import__('traceback... [18:39:10] 06Labs, 10Tool-Labs, 10video2commons: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2900966 (10yuvipanda) Interesting - on a local install, I can do: ``` >>> os.stat('šabc') posix.stat_res... [18:39:28] 06Labs, 10Tool-Labs, 10video2commons: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2900967 (10yuvipanda) Also, https://uwsgi-docs.readthedocs.io/en/latest/StaticFiles.html might be a more... [18:43:08] yuvipanda: uhoh [18:43:14] thx [18:43:54] 06Labs, 10Tool-Labs, 10video2commons: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2900968 (10yuvipanda) When I run the same code inside a python2 kubernetes container, it seems to work fi... [18:44:02] Steinsplitter: yw! [18:54:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:00:33] 06Labs, 10Tool-Labs, 10video2commons: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2900985 (10zhuyifei1999) The filename is in unicode, not string. [[http://stackoverflow.com/questions/28... [19:09:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:10:56] 06Labs, 10Tool-Labs, 10video2commons: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2900989 (10zhuyifei1999) http://askubuntu.com/questions/162391/how-do-i-fix-my-locale-issue says to run `... [19:17:04] 06Labs, 10Tool-Labs, 10video2commons: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2901005 (10zhuyifei1999) >>! In T154088#2900967, @yuvipanda wrote: > Also, https://uwsgi-docs.readthedocs... [19:18:32] zhuyifei1999_: around? [19:18:42] yeah [19:19:19] * zhuyifei1999_ hates UnicodeErrors [19:19:22] zhuyifei1999_: so using uwsgi directly is going to give you much better perf esp for large files, since it's just C code without a python layer. but I'm going to fix the locale issue too [19:20:02] well, I'm pretty sure the python stuff go through sendfile() syscall [19:20:26] so those sending part is handled by kernel [19:20:42] does sendfile support NFS? [19:21:05] and hih-level stuffs are generally more feature rich and low-level equilivants [19:21:09] why not? [19:21:54] *equivalents (I think this is how to spell this thing) [19:22:54] hmm, I remember reading sendfile doesn't support NFS but can't find anything to back it up now :) [19:23:03] anyway, if you don't see a perf problem I guess it's simpler to keep it all in flask [19:23:11] (and I agree on uwsgi's options being scary) [19:23:51] lol [19:24:38] fwiw, https://github.com/pallets/flask/blob/master/flask/helpers.py#L670 <= this is where 400 was raised [19:24:55] My task is stuck on qw. Is there a deployment now or? [19:34:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:07:34] zhuyifei1999_: can you do a 'webservice restart'? [20:07:41] k [20:09:08] yep it works https://tools.wmflabs.org/video2commons-test/static/ssu/Prvi_%C5%A1olski_dan_na_Ravnah_in_Prevaljah_-_2015.webm [20:09:31] zhuyifei1999_: \o/ awesome! [20:09:37] I'll merge the patch and rebuild all other containers [20:09:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:12:21] zhuyifei1999_: can I close the bug? [20:12:32] sure [20:12:48] thx again :) [20:15:25] zhuyifei1999_: np! thanks for digging in and finding the root cause! [20:15:52] 06Labs, 10Tool-Labs, 10video2commons, 13Patch-For-Review: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2901034 (10yuvipanda) 05Open>03Resolved a:03yuvipanda Fixed! If your tool / w... [20:16:10] np :) [20:17:02] 06Labs, 10Tool-Labs, 10video2commons, 13Patch-For-Review: Tool labs uwsgi + flask seems unable serve static files with non-ascii characters in filename, generating 400 Bad Request - https://phabricator.wikimedia.org/T154088#2901037 (10yuvipanda) As a note, the reason this was unspotted was because we had:... [20:18:30] * zhuyifei1999_ gtg [21:44:50] 10Wikibugs: wikibugs showing jenkins merges for user L10n-bot - https://phabricator.wikimedia.org/T154094#2901075 (10Paladox) [21:45:48] 10Wikibugs: wikibugs showing jenkins postmerge events for user L10n-bot - https://phabricator.wikimedia.org/T154094#2901087 (10Legoktm) [22:05:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:10:44] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:48:51] anyone from labs here? [23:49:12] trusty is stucking in qw states [23:49:33] from labs? [23:49:55] tools.giftbot [23:50:15] 698813 0.30004 rue tools.giftbo qw 12/24/2016 23:01:03 1 [23:50:18] 698815 0.30004 picdwba tools.giftbo qw 12/24/2016 23:01:03 1 [23:50:21] 698877 0.30003 sg tools.giftbo qw 12/24/2016 23:07:03 1 [23:50:24] 699313 0.30001 daysection tools.giftbo qw 12/24/2016 23:40:58 1 [23:50:34] has trusty an overload? [23:51:27] Krenair: do you know anything about it? [23:51:32] no [23:51:45] Krenair: what can I do now? [23:51:45] I know relatively little about the tools systems [23:52:03] yuvipanda: ^ [23:54:15] doctaxon, tell me more about the issue [23:55:07] are trusty jobs simply not working? or...? [23:55:46] new jobs are stucking in qw state in qstat [23:57:47] how serious is this problem for users? [23:58:40] seems that no new bot job or tool works, that's started since then [23:58:57] I'm guessing this breaks a lot of cron jobs etc. [23:59:09] ya [23:59:10] this is not happening with precise jobs? [23:59:24] I'm seeing if I can get hold of one of the labs ops team