[00:21:58] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c10 (10Bryan Davis) a:5Bryan Davis>3None Un-licking this cookie. I took a shot at it but the problem seems to be deeper in the stack. [00:41:40] Is there any chance of getting Flickr::API perl module installed ? [00:43:38] Needed to add a running count of 'map' tags on this page for an event and campaign tomorrow: https://commons.wikimedia.org/wiki/Commons:British_Library/Mechanical_Curator_collection/map_tag_status [00:44:09] or rather: later today (UTC) [00:44:11] 3Wikimedia Labs / 3tools: program created by proprietary compiler allowed on labs? - 10https://bugzilla.wikimedia.org/72253#c6 (10Luis Villa (WMF Legal)) This is the kind of thing where the open source definition's descent from the Debian Free Software Guidelines shows through. That language makes more sense... [01:03:42] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c11 (10Antoine "hashar" Musso (WMF)) I forgot to update this bug after my debugging session on Oct 24th here is the rough brain dump: The Gearman... [01:37:14] 3Wikimedia Labs / 3tools: Request for the Flickr::API perl module to be installed on toollabs - 10https://bugzilla.wikimedia.org/72800 (10James Heald) 3UNCO p:3Unprio s:3enhanc a:3Marc A. Pelletier Intention: Wanted for a campaign to use indexes on Commons to find maps in the 1-million image British... [01:41:11] 3Wikimedia Labs / 3tools: Request for the Flickr::API perl module to be installed on toollabs - 10https://bugzilla.wikimedia.org/72800#c1 (10James Heald) I should add that the campaign starts with a tagathon event in eight hours time ... so I'm possibly being optimistic. Just to clarify: with the Flickr::AP... [07:05:47] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-dev.diskspace._var.byte_avail.value (11.11%) [09:27:02] PROBLEM - ToolLabs: Low disk space on / on labmon1001 is CRITICAL: CRITICAL: tools.tools-login.diskspace.root.byte_avail.value (11.11%) [09:35:16] RECOVERY - ToolLabs: Low disk space on / on labmon1001 is OK: OK: All targets OK [12:15:06] Hi all! I have just started yesterday my crontab with "jsub -N sign-of-life ...sign-of-life.sh > /dev/null" avoid to spamming admins. Today I have received a mail two times from Cron Daemon; "/bin/sh: 1: cannot create /dev/null, : Permission denied" [12:15:16] Is it normal? :) [12:18:49] ato_: better to redirect error output to /dev/null [12:18:55] Note the comma in your copied eror message. [12:19:09] You seem to have tried to create /dev/null, <-- stray comma [12:20:17] There was no "," in there. [12:20:36] The mail was only two lines. [12:20:36] ato_: replace "> /dev/null" with "> /dev/null 2>&1" [12:21:10] Thanks, mate! I will try it. [12:30:50] ato_: from rereading your error message, I guess might something else is wrong [12:31:46] Could be. [12:31:58] I never ran a jsub job (I my last experience was with crontab 4 years ago...) so I guess https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#jsub_options would be useful [12:32:24] *and my [12:33:43] I think I will try jsub with -quiet option. :) [13:35:29] 3Wikimedia Labs / 3tools: webservice truncates files - 10https://bugzilla.wikimedia.org/72813 (10Magnus Manske) 3NEW p:3Unprio s:3major a:3Marc A. Pelletier I have a file at /data/project/wikidata-todo/public_html/duplicate_sitelinks.html which is exposed as https://tools.wmflabs.org/wikidata-todo... [13:36:26] 3Wikimedia Labs / 3tools: webservice truncates files - 10https://bugzilla.wikimedia.org/72813 (10Magnus Manske) [13:48:24] YuviPanda: Ping? [13:48:29] 3Wikimedia Labs / 3tools: webservice truncates files - 10https://bugzilla.wikimedia.org/72813#c1 (10Marc A. Pelletier) 5NEW>3ASSI p:5Unprio>3High a:5Marc A. Pelletier>3Yuvi Panda Verified that lighttpd correctly sends out the whole file, but that the /proxy/ stalls after some variable amount of d... [13:48:44] YuviPanda: https://bugzilla.wikimedia.org/show_bug.cgi?id=72813 is a big one; the proxy mangles large static files seemingly. [14:00:47] Coren: hmm, ok. I'm off until Monday, will look then [14:01:20] Hm. I'll try to figure it out then because that's a major bug. [14:14:27] 3Wikimedia Labs / 3tools: webservice truncates files - 10https://bugzilla.wikimedia.org/72813#c2 (10Magnus Manske) Now it works. Did someone fix the proxy? [15:50:12] 3Wikimedia Labs / 3tools: webservice truncates files - 10https://bugzilla.wikimedia.org/72813#c3 (10Marc A. Pelletier) Yuvi has, as far as I know, left for the weekend; but it wouldn't be the first time that a member of the operations team fixed an important bug during their "time off". I'm leaving the bug... [21:54:42] 3Wikimedia Labs / 3tools: webservice truncates files - 10https://bugzilla.wikimedia.org/72813#c4 (10Yuvi Panda) I swear I didn't touch anything! [21:55:30] Coren: I didn't touch it! [21:55:37] Coren: I wass off any computing device until now [22:06:52] Anyone with just a little fcgi on labs experience around? Really couldn't run even a little sample one also [22:44:22] ebraminio: some, yes. [22:45:00] ebraminio: did you follow https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Example_configurations ? [22:57:09] PROBLEM - ToolLabs: Puppet freshness check on labmon1001 is CRITICAL: CRITICAL: tools.tools-dev.puppetagent.time_since_last_run.value (22.22%)