503 Service Unavailable

[00:34:42] 3Wikimedia Labs / 3Infrastructure: mediawiki_singlenode's $wgMainCacheType = CACHE_ACCEL can't be overridden - 10https://bugzilla.wikimedia.org/72600 (10spage) 3NEW p:3Unprio s:3major a:3None Puppet change Ia07c9f7c00960fecc0765c7c13e3897981c69600 added # Cache where it's easy to $wgMainCacheTyp... [00:35:38] hey, a puppet change is breaking our ee-flow labs instance (bug 72600). Is there a way to disable puppet updates on a particular labs instance? [00:35:55] spagewmf: puppet agent --disable [00:36:03] but you should blame that change instead :) [00:37:02] we have monitoring for puppet breakage etc [00:37:05] it will trigger [00:37:15] should have a real fix instead of disabling it [00:38:04] mutante: I'll attempt a real fix [00:38:31] cool! [00:43:01] https://gerrit.wikimedia.org/r/#/c/169288/ [01:47:52] 3Wikimedia Labs / 3wikistats: Make a stats table for Wikivoyage - 10https://bugzilla.wikimedia.org/44194#c11 (10Daniel Zahn) 5PATC>3RESO/FIX thanks for pointing it out (and using the bug). for some reason there was simply no cron job defined. running the update manually worked just fine. sudo -u wikista... [02:35:28] (03PS1) 10Legoktm: Refactor into a proper class [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/169309 [02:36:46] (03CR) 10Legoktm: [C: 032] Fetch list of extensions from API list=extdistrepos [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/168819 (owner: 10Legoktm) [07:57:31] Some help would be great... [07:58:08] ...to get back my rights in my project directory. [11:07:55] Is there a way I can make a script ? [11:23:16] Coren: pythongrid wants libdrmaa.so, so I take it this drmaa thing isn't installed on the labs? [12:16:53] hi what does Dr state mean for a job in tools-lab? [12:17:04] dr [12:18:38] rohit-dua: Disabled running. [12:19:23] w930913: so disabled means that its running but really doing anything? [12:21:23] I think it means that the running has been disabled. I.e. it's not running. [12:23:19] rohit-dua: I think I've seen it go through that state as it initialises, and I think it might end up in that state if certain things go wrong. [12:24:17] w930913: can I restart the job somehow so that the dr changes to r.? [14:07:48] w930913: I don't think that it is. Bz it and we can add it. [14:42:28] goddamit, I need to get something swatted and have spotty internet... [14:48:17] andrewbogott: around? [14:48:58] !ping [14:48:58] !pong [14:54:49] YuviPanda: I'm here [14:55:13] andrewbogott: cool! I'm setting up SWAT today, in about 5 minutes :) will you be around to run sync? [14:55:18] yep [14:55:23] andrewbogott: cool, thanks! [14:58:50] andrewbogott: I'm having a slightly hard time with bandwidth, can you help do submodule bump patches? [14:59:44] YuviPanda: maybe? I'm not sure I've done that before. [14:59:53] What branches do we need it on? [15:00:15] andrewbogott: 1.25wmf4 and 1.25wmf5 [15:00:25] andrewbogott: the backported patches are in https://wikitech.wikimedia.org/wiki/Deployments#Near-term [15:00:49] Is the backport anything more than a cherry-pick? [15:01:00] andrewbogott: nope [15:05:45] !ping [15:05:45] !pong [15:05:51] baaahhhh [15:06:07] SO MUCH LAG [15:06:27] YuviPanda: is there any reason I shouldn't just set both of those core branches to OSM head? [15:06:46] andrewbogott: nope, nothing I can think of [15:06:52] andrewbogott: there's been no other patches [15:07:05] andrewbogott: wait, [15:07:08] andrewbogott: that will break things [15:07:18] ? [15:07:25] andrewbogott: ask manybubbles? I think you need to set them to HEAD in their respective branches [15:07:40] andrewbogott: so core's 1.25wmf5 should be OSM's 1.25wmf5 and so on [15:07:45] I don't think there are branches in OSM [15:07:55] andrewbogott: there are! [15:07:58] andrewbogott: auto created [15:08:05] YuviPanda: yeah - set them to the 1.25wmf5 branch, etc [15:08:08] andrewbogott: we just cherry picked and merged the branches. [15:08:35] OpenStackManager$ git checkout -b wmf/1.23wmf4 wmf/1.23wmf4 [15:08:35] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (10.00%) WARN: tools.tools-dev.diskspace._var.byte_avail.value (100.00%) [15:08:36] fatal: git checkout: updating paths is incompatible with switching branches. [15:08:36] Did you intend to checkout 'wmf/1.23wmf4' which can not be resolved as commit? [15:08:42] So what's it called then? [15:09:15] andrewbogott: https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Updating_the_submodule [15:13:15] That's what I'm saying, though... [15:13:16] OpenStackManager$ git checkout origin/wmf/1.23wmf4 [15:13:16] error: pathspec 'origin/wmf/1.23wmf4' did not match any file(s) known to git. [15:13:34] no branches in OSM [15:13:55] manybubbles, YuviPanda ^ [15:14:21] andrewbogott: git fetch first? [15:14:26] just grasping at straws [15:14:38] I fetched [15:14:49] andrewbogott: git fetch -a lists remote branches for me... [15:15:02] yeah, and there aren't any [15:15:18] YuviPanda: want me to just +2/+2 those changes and submit them and ignore zuul? [15:15:23] andrewbogott: hmm, my fetch just completed, and I have 'em... [15:15:30] manybubbles: yeah [15:15:42] manybubbles: I foresee a few hours of ignoring Zuul :) [15:15:58] merged [15:16:01] andrewbogott: let me try... [15:16:10] !ping [15:16:10] !pong [15:16:27] goddamn, shitty spotty internet [15:16:27] * YuviPanda curses self [15:16:38] bah, I need to fetch again.... [15:17:37] !ping [15:17:37] !pong [15:31:20] !ping [15:31:20] !pong [15:33:33] !ping [15:33:33] !pong [15:39:45] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (10.00%) WARN: tools.tools-dev.diskspace._var.byte_avail.value (100.00%) [15:39:58] !ping [15:39:58] !pong [15:39:58] !ping [15:39:58] !pong [16:18:01] Hi, could anyone help to make backwards the command "chmod -R -g+rw ./scripts" ? [16:26:43] ato-: Do you not have access as the owner of those files? Do you need someone with root access to help? [16:52:31] bd808:Yes. [16:53:06] I think I need someone with root access. [16:53:14] ato-: andrewbogott or YuviPanda may be able to help you. Tell them which directory and host you need a hand on. [16:53:30] Thank you! [16:54:57] ato-: heya! where are the files? [16:55:15] ato-: is this toollabs? [16:56:08] Yes [16:56:15] ato-: where are the files? [16:56:49] . [16:57:07] -/data/project/ato/scripts [16:58:55] ato-: moment [17:00:18] Thank you, mate. [17:02:32] ato-: try now? [17:02:48] :) [17:03:27] Great [17:03:35] ato-: :) cool! [17:03:37] * YuviPanda goes to eat some food [17:03:41] Can I wish more? [17:04:05] ato-: hmm? [17:04:14] Thank you anyway [17:04:21] :) yw! [17:04:46] I made a second project called ato-01 [17:04:55] it could be deleted [17:05:27] It could be done only an admin,. [17:18:30] ato-: I can delete ato-01 if you're sure that's safe [17:19:05] Hm, I see 'Ato 01' and 'tools.ato' and 'tools.ato-01' [17:27:08] andrewbogott: Yes, please [17:27:19] ato-: which of those three shall I delete? [17:28:19] please all the ato-01 project to delete if it is possible [17:28:39] should be empty [17:29:25] tools.ato-01? [17:29:35] yes please [17:29:46] not my user ato-01 [17:29:47] :) [17:30:17] ok, done [17:30:30] thank you [17:31:10] now only what I need, get my crontab to work :) [17:32:38] But I think I should make it alone.:) [18:40:37] 3Wikimedia Labs / 3deployment-prep (beta): SpecialCite's i18n is still being loaded which is breaking CiteThisPage in BetaLabs - 10https://bugzilla.wikimedia.org/71112#c6 (10James Forrester) 5PATC>3RESO/FIX Now fixed in Beta Labs. [19:38:26] YuviPanda or Coren: Do you want to take a shot at rescuing puppet on wikitech-test-frontend? I've been going in circles for a while. It should be self-hosted but is rejecting its own key [19:41:05] andrewbogott: I can take a look in ~30m if you want. [19:41:19] Coren: sure… I'll take one more stab in the meantime [20:28:17] Coren: I may have this partially sorted now [20:28:27] What was the issue? [20:28:39] Also, I'm free if you still need help. [20:34:13] andrewbogott: ^^ [20:34:52] Coren: Just a failure of self-hosted puppet gymnastics. I think it's straightened out now. [21:03:18] What the .... [21:03:22] I have a phantom instance in deployment-prep [21:03:24] https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000084.eqiad.wmflabs [21:03:34] It's in the list of instances, and there is in fact an instance responding to ping at that IP [21:03:38] But SSH connections time out [21:03:53] I tried to reboot it, then I tried to delete it, and in both cases I get an error saying the instance doesn't exist [21:04:24] Also https://wikitech.wikimedia.org/wiki/Special:NovaInstance is broken and doesn't list any instances for deployment-prep at all [21:05:01] OK, egg on my face [21:05:05] I had to log out and back in again [21:05:10] But the error messages are *very* confusing [21:06:00] !log deployment-prep Rebooting deployment-parsoid04, wasn't responding to ssh [21:06:06] Logged the message, Mr. Obvious [21:09:55] RoanKattouw: They are super confusing. I have learned that the fix for many mystery wikitech errors is to log out and log back in. :( [21:30:01] That mediawiki/openstack integration thing is an ill-conceived nightmare. [21:30:14] Which is why replacing it is on our roadmap. [21:34:47] Hi (from #wikimedia-qa) Parsoid still broken on beta labs [21:34:55] VE Edit beta gives dialog 503 Service UnavailableError loading data from server: parsoidserver-http-bad-status: 503: [21:35:56] RoanKattouw is this related to your "Rebooting deployment-parsoid04" ? [21:36:03] Yes [21:36:06] I'm still working on that problem [21:36:14] k thanks! [21:36:36] !log deployment-prep Creating deployment-parsoid05 as a replacement for the totally broken deployment-parsoid04 (also as a trusty instance rather than precise) [21:36:38] Logged the message, Mr. Obvious [22:07:04] Is it expected that I can't upload images to wikitech? Does commons really want all my cluster diagram sketches? [22:07:18] bd808: they do :) [22:07:26] ok [22:12:54] legoktm: I think it's just a misconfiguration. I can find lots of locally uploaded files on wikitech. [22:13:33] we recently turned off uploads on all cluster wikis that didn't have a local MediaWiki:Licenses [22:13:48] yeah and nobody has set that up on wikitech [22:13:52] https://wikitech.wikimedia.org/wiki/MediaWiki:Licenses [22:14:06] uploading to commons is still preferred IMO [22:14:19] * bd808 shrugs [22:15:08] what category, etc? I don't want to have my images for the logstash page on wikitech deleted randomly [22:17:03] bd808: https://commons.wikimedia.org/wiki/Category:Wikimedia_technical_operations [22:17:10] thanks [22:17:12] bd808: it won't get deleted if you license it properly :) [22:24:20] hi I have some python bot code that I want to run everyday at 19:45 [22:25:12] if I understand correctly I have to store that code on my wikitech account? [22:28:23] hello? [22:29:25] lbertolotti: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Submitting.2C_managing_and_scheduling_jobs_on_the_grid may help you [22:29:54] * bd808 doesn't really use tool labs [22:30:46] lbertolotti: Even more specifically -- https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Scheduling_jobs_at_regular_intervals_with_cron [22:32:16] yes that I understand [22:32:19] Tools and bot code should be stored in your tools account, where it can be managed by multiple users and accessed by all execution hosts. Specific information about configuring web services and bots, along with information about licensing, package installation, and shared code storage, is available at the #Developing on Tool Labs section. [22:35:04] I'm not sure I understand how to do that [22:36:15] lbertolotti: where did you get lost? Do you have a wikitech account? [22:36:22] yes [22:36:33] with the same nick I'm using here [22:37:56] Did you join the tools project? [22:40:25] bd808 yes [22:41:17] lbertolotti: I just gave you shell rights, and added you to tools. You were not a member until 10 seconds ago [22:41:28] heh [22:41:47] lbertolotti: if you upload a key to wikitech you should be able to log in to tools-login.wmflabs.org [22:42:35] you mean a linux key? [22:43:39] lbertolotti: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Generating_and_uploading_an_SSH_key [22:43:49] You'll need to create an ssh keypair and then upload the public key to wikitech [22:44:06] …in order to get access to anything [22:45:08] the link tells me to run ssh-keygen -t rsa [22:46:11] lbertolotti: yes, if you don't already have a preferred keypair [22:46:35] my terminal is asking for a passphrase [22:47:48] well I just went for empyt passphrase [22:49:38] ok I have a file with a huge sequence of number and letters [22:50:23] lbertolotti: You have to be careful with empty passphrases, it means that anyone who gets a hold of your private key can pass off as you. [22:50:39] oh I see [22:50:50] better change that then [22:52:10] where's the link to upload the public key? [22:52:27] It's in your preferences, on Wikitech. Check the last tab. [22:54:47] ok uploaded the key [22:54:56] how do I know if that worked? [23:07:56] ok now I have service group [23:08:08] lbertolotti: If you're able to log in to tools-login, it worked! [23:08:36] I'm logged at wikitech [23:09:45] to work on a tool you'll need to connect via ssh to an actual tools host. Wikitech is just the web interface for managing users and such. [23:09:49] And documentation. [23:11:14] permission denied [23:11:26] I just did ssh yourshellaccountname@tools-login.wmflabs.org [23:11:50] lbertolotti: ssh-add -l ? does it show your key being loaded? [23:12:00] lbertolotti: what is your shell name? [23:13:09] ssh-add -l gives me a alfanumeric sequence [23:13:30] lbertolotti: that sounds like it should be [23:13:55] as long as there aren't multiple different keys [23:14:29] what do u mean? [23:14:40] I generated a private key and a public key [23:15:07] using ssh-keygen -t rsa [23:15:20] lbertolotti: ssh-add -l lists all your keys. Presumably in this case 'all' was just one key. [23:15:23] What is your shell name? [23:16:14] lbertolotti: yea, counting each pair of private/public as 1 [23:16:34] what you did sounds right [23:16:48] I think the shell name is lbertolotti [23:17:21] you mean should be here https://wikitech.wikimedia.org/wiki/Category:Shell_Access_Requests ? [23:17:29] yeah, looks right. Can you try to ssh again? I'll look at the logs. [23:18:35] looks good to me [23:19:00] is this the starting page? [23:19:01] ########## Wikimedia Tool Labs ##### ##### #### ###### #### This is a server of the tools project, the home *### ############ ###* of community-maintained external tools supporting # #####"""".>##### # Wikimedia projects and their users. #. ####( \ ######## .# ## #####) ######### ## ## ####' """### ## ## ### # ### #<<### ## Use of this syst [23:19:30] yep, you logged in [23:20:04] ok so now I have this bot.py file in my hd [23:20:16] written in python of course [23:20:30] where do I upload the code? [23:21:01] lbertolotti: you can use "scp" [23:21:26] have you copied files on the shell before? [23:22:01] scp is like cp, just that you can do it between multiple computers, via ssh [23:23:20] usage: scp [-12346BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file] [-l limit] [-o ssh_option] [-P port] [-S program] [[user@]host1:]file1 ... [[user@]host2:]file2 [23:25:38] lbertolotti: I need to go, but it sounds like at least all of your access issues are resolved. Other folks should be able to help you in this channel (although it's generally busier during european daylight hours) [23:26:05] where do I specify the folder path to file? [23:30:00] to run a python code every day do I have to use jsub or jstart? [23:30:39] hello? [23:31:43] anyone knows how to run python code in tool labs? [23:45:00] 0 22,23 * * * [ "$(TZ=:Europe/Berlin date +\%H)" = "00" ] && jsub is too complicated [23:45:16] I just need to run the job every day at 19:45 utc time [23:45:56] 45 19 * * * then [23:46:26] !log deployment-prep Updating puppet repo on deployment-salt puppet master [23:46:28] Logged the message, Mr. Obvious [23:46:29] yeah but that command does a lot more stuff [23:46:56] don't worry about it [23:47:02] The above crontab line instructs the system to check on 22:00 UTC (23:00 CET and 0:00 CEST) and 23:00 UTC (0:00 CET and 1:00 CEST) whether it is midnight in Berlin, and if so, calls jsub. Note that you can't just replace "Berlin" with "Hamburg"; the values for TZ are limited to those found at /usr/share/zoneinfo. [23:47:10] RoanKattouw: Did it get stuck on a merge conflict? [23:47:16] !log deployment-prep ...which was a no-op [23:47:18] Logged the message, Mr. Obvious [23:47:24] bd808: No, Varnish manifests are broken for beta [23:47:29] Error: Failed to apply catalog: Could not find dependency File[/usr/lib/ganglia/python_modules] for File[/usr/lib/ganglia/python_modules/varnish.py] at /etc/puppet/modules/varnish/manifests/monitoring/ganglia.pp:10 [23:47:50] Hmm [23:47:54] I guess I can exploit this [23:47:54] hmmm... do we still have a local varnish patch? [23:48:15] I need to do https://gerrit.wikimedia.org/r/169622 [23:48:24] But if puppet doesn't run, then I can just change the config locally :P [23:48:26] oh ganglia crap :( [23:48:36] (I'll also apply it as a local cherry-pick just in case puppet is fixed later)