[00:06:38] does tools labs has pywikibot library by default [00:06:40] ? [00:11:14] i think so, but i don't know how to use it [00:12:29] gifti https://wikitech.wikimedia.org/wiki/User_talk:Tim_Landscheidt#Python_code [00:17:48] lbertolotti: have you read https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Pywikibot? [00:18:52] become toolname become: no such tool 'toolname' [00:20:56] hello, someone there? [00:21:39] lbertolotti: maybe create a tool then first [00:22:01] !ask [00:22:01] Hi, how can we help you? Just ask your question. [00:24:52] just a quick question: i would like to study and then contribute to the infrastructure part of wikipedia (puppet and stuff connected) which name has the related project? [00:25:02] rl: operations/puppet [00:25:04] gifti well I'm using source .bash_profile [00:25:14] gifti: well I'm using source .bash_profile [00:25:34] gifit: I mean tools.portalbox@tools-login:~$ [00:25:40] rl: https://gerrit.wikimedia.org/r/#/q/project:operations/puppet,n,z [00:26:48] rl: git clone https://gerrit.wikimedia.org/r/p/operations/puppet [00:26:48] ty [00:26:58] yw [00:27:08] rl: and please ask questions, etc.! [00:27:35] rl: is useful, too. [00:27:41] gifti: I keep getting tools.portalbox@tools-login:~$ source .bash_profile -bash: .bash_profile: No such file or directory [00:28:47] well, step 2 is: create that file [00:31:36] gifti: already did that [00:33:09] lbertolotti: i don't see it in ~tools.portalbox [00:35:01] gifti: don't see what? [00:35:09] the instruction was : In your home directory, create (or edit, if it exists already) a ‘.bash_profile’ file to include the following line. The path should be on one line, though it may appear to be on multiple lines depending on your screen width. When you save the .bash_profile file, your settings will be updated for all future shell sessions: [00:35:46] there is no ~tools.portalbox/.bash_profile [00:40:11] gifti: were are you seeing that? [00:41:07] i just did 'ls -a ~tools.portalbox' [00:44:10] gifti: try now [00:45:54] lbertolotti: now you should be fine [00:49:25] gifit: don't understand [00:49:35] tool labs keeps asking [00:49:35] Your default user directory is "/data/project/portalbox" Do you want to use that directory? ([y]es, [N]o) yes New user directory? [00:50:42] i have no idea where that comes from? [00:51:53] gifiti:it's step 5 [00:52:48] well then i guess you want to use that directory [00:54:15] gifit: yes but why does tool labs asks for new user directory? [00:54:59] well, it's a configuration script [01:04:39] gifti: well I just left that empty and it worked [01:05:20] gifti: so let me see if I understand this I need to copy the python code to the portalbox folder in tool la bs? [01:05:46] i guess so, yes [01:09:34] gifti: do u use linux= [01:09:37] ? [01:12:21] ok I think I have a workaround [01:13:30] i do [01:14:25] gifti: well what I have is this [01:14:40] tools.portalbox@tools-login:~$ dir apicache bot.py cgi-bin n public_html python2.err python2.out replica.my.cnf throttle.ctrl user-config.py user-fixes.py [01:17:30] and? [01:20:50] gifti: I was just doing jsub python bot.py [01:22:38] gifti: now it worked! https://en.wikipedia.org/w/index.php?title=Portal:Business_and_economics/Market_Indices&action=history [01:22:44] woo [01:23:32] gifti: so the thing is u see this code needs to be run everyday at 19:45 [01:25:03] gifti: If I do 45 19 * * * [ "$(TZ=:Europe/Berlin date +\%H)" = "00" ] && jsub python bot.py [01:25:24] gifiti: will tool labs remember to do this everyday? [01:25:52] the [ ... ] && part is not needed [01:26:35] gifti: also I think I need to make the edit using this https://en.wikipedia.org/wiki/User:Portal_box_bot [01:26:42] it would only stop it from workibg in this case [01:28:13] gifti: hum so just using my regular account is fine? [01:29:43] you could change the user setting [01:30:27] gifti: where? [01:31:05] one part is in user-config.py i guess [01:31:19] i don't know where the passwords are [01:38:05] gifti:well perhaps I need to get the bot approved first [01:39:55] if you make one edit per day in portal namespace i don't recall that you need approval for that [01:41:44] gifti: yeah well regarding that [01:41:54] gifti: I think the syntax is wrong [01:41:55] tools.portalbox@tools-login:~$ 45 19 * * * jsub python bot.py 45: command not found [01:42:39] well, you manipulate your crontab via the crontab command [01:43:51] gifti: I inputed crontab but now the terminal is waiting for something [01:45:06] the easiest thing is to write it into a file named crontab and then to invoke 'crontab crontab' [01:51:36] gifti: I've got this Edit this file to introduce tasks to be run by cron. # # Each task to run has to be defined through a single line # indicating with different fields when the task will be run # and what command to run for the task # # To define the time you can provide concrete values for # minute (m), hour (h), day of month (dom), month (mon), # and day of week (dow) or use '*' in these fields (for 'any').# # Notice t [01:53:19] well, add your line to that, save and exit, you can also delete the given text [01:58:25] gifti: I'm sorry what line? [01:59:34] 45 19 * * * jsub python bot.py [02:01:06] gifti: all right how do I know if that worked? [02:01:25] gifti: I have to wait until tomorrow? [02:01:38] basically [02:02:14] you can also try a different time, like 5 2 * * * [02:02:32] gifit: well let me see then [02:02:36] but it should work [02:03:06] gifti: how does crontab knows it's UTC time? [02:03:42] because all labs instances run with utc [02:05:21] gifti:yeah well that worked [02:05:34] yay [02:06:31] gifti: I mean tools.portalbox@tools-login:~$ qstat job-ID prior name user state submit/start at queue slots ja-task-ID ----------------------------------------------------------------------------------------------------------------- 5192950 0.30000 python2 tools.portal r 10/29/2014 02:05:06 task@tools-exec-06.eqiad.wmfla 1 [02:06:47] gifti: still need to get the bot approved at wikipedia [02:07:25] if you think so [02:08:14] gifti: well talking about timezone [02:08:25] gifiti: what time is it where you are now? [02:08:44] 3:08 cet [02:09:50] gifti: alright then good night to you and thanks for your help [02:10:19] yw :) [06:35:53] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [13:31:07] Hi, I am just starting the tool labs, and I don't understand how to run a CGI script [13:31:33] I created it with read and execute access for all users: -rwxr-xr-x 1 tools.projetpp tools.projetpp 142 Oct 29 13:25 /data/project/projetpp/cgi-bin/core.py [13:31:47] but it does not seem to be found: http://tools.wmflabs.org/projetpp/cgi-bin/core.py [13:32:59] is there anything I am doing wrong? [13:33:26] pinkieval: have you tried running 'webservice start' as your tool? [13:33:33] yes [13:34:16] pinkieval: ah, hmm. I'm not sure :| tried reading through https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Web_services? [13:34:22] * YuviPanda isn't around too much atm, will have to go soon [13:35:34] I already did, but I can try it again [13:36:27] ok! [13:37:17] copy-pasting the default config gives an entry in the error log: 2014-10-29 13:36:06: (configfile.c.912) source: /var/run/lighttpd/projetpp.conf line: 554 pos: 12 parser failed somehow near here: (EOL) [13:38:40] and I don't see obvious copy-pasting issue in the file [13:38:57] pinkieval: ah, hmm. add a newline at the end of the file? [13:39:10] of your file, I mean. [13:39:32] no change [13:40:01] after a webservice start as well? [13:40:03] (even after restart of the webservice, of course) [13:40:12] what's your tool name? [13:40:27] projetpp [13:45:57] pinkieval: oh wait, where did you copy paste it from? [13:46:02] the wiki [13:46:18] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Default_configuration [13:47:47] ah, hmm [13:47:50] pinkieval: I'm not sure, sorry :( [13:47:52] and I've to go [13:48:04] pinkieval: I don't think you should clone the default configuration in your .lighttpd.conf but just overrride the parameters you want [13:48:39] I shouldn't get a syntax error then [13:49:14] pinkieval: I belive also that you should use ~/public_html for reachable from the web scripts [13:49:46] what is the point of cgi-bin then? [13:50:30] I don't know [13:50:35] putting the .py file in public_html makes it downloadable [13:51:22] the "Apache-like cgi-bin directory" section of the doc is maybe what you are looking for [13:52:44] that's what I'm testing right now [13:54:01] what [13:54:09] it's not actually downloadable [13:54:20] I can download a file, but it's empty [13:54:31] and I get this in the error log: [13:54:33] File "/data/project/projetpp/public_html//cgi-bin/core.py", line 2, in [13:54:34] from flup.server.fcgi import WSGIServer [13:54:35] ImportError: No module named flup.server.fcgi [13:55:01] flup.server.fcgi is said to be installed [13:56:53] maybe because of the "//" in the path or something silly like that [13:58:49] got it [13:58:57] it is only installed for python 2… [15:17:29] Hi. Is someone around with access to others tools (root?)? I made a localization patch which is *very important* for making commonshelper works, I emailed magnus and even put a message on his talk page and he didn't reply for a month. The patch http://pastebin.com/NTgF4RgS and the file is located on /data/project/commonshelper/public_html/ [15:18:14] Coren: Can you have a look ^ [15:20:10] It's... really not apropriate for me to patch someone else's tool ebraminio. Isn't the commonshelper in a git repo where you could make a pull request? [15:21:38] ebraminio: looks like most of his code is on bitbucket https://bitbucket.org/magnusmanske [15:21:57] but it doesn't look like commonshelper is there... I'd ping him there and ask him to add it, then you can make a pull request [15:22:24] Coren: No, it is commonshelper2 which have a git clone and it is using configuration from meta, but while transferring to labs it got stopped. The tool (commonshelper) just have one maintainer and he is reponding to his email or talk page. I think this is the point of making tools separate from accounts [15:22:35] how can I get the exact file name from a File:XYZ page with the API? [15:22:54] (I'm sorry; I know it could be helpful, but having the sysadmins edit live code is just begging for trouble even if nothing breaks because there is no change history) [15:23:37] ebraminio: Well, taking over a tool is possible though it might be simplest to clone it. If you want, I can start the ball rolling to add you as a maintainer of the tool. [15:23:40] Nettrom: there is not any, I checked, there is not even a .git folder on that folders [15:25:18] But yeah, I've had trouble reaching Magnus myself for a bit so I'm not surprised you have. [15:25:19] It is fairly simple patch, http://pastebin.com/NTgF4RgS local users are asking me for fixing it [15:27:16] ebraminio: It wouldn't be right for me to apply it even if it was a single whitespace change. Lemme look into something, I'll get back to you shortly. [15:31:17] Coren: Ok. Thank you [16:26:58] Coren: "fork a tool" sounds like a new feature to implement [16:28:52] petan: sounds like 'don't do that' to me [16:29:08] sounds like "do that" to me [16:29:22] users would need to mark their tools as forkable [16:29:22] splitting communities is bad, etc. [16:29:51] if people could explicitly flag their tools as designed for forking this feature would work just fine [16:29:55] oh, a 'development fork' kind of thing would make sense, I guess [16:30:14] still makes me wary of getting 15 edit counters that are all the same-but-not-quite [16:30:35] lot of people were like you when Linus started his work on git [16:30:53] yet it became most popular versioning sw [16:31:31] it would ultimately solve bus factor issue for sure [16:31:38] why would Labs reimplement something that Github and Bitbucket already has? [16:31:42] more maintainers? yes. [16:31:51] multiple deployments of the same project? ....not really [16:32:00] Nettrom: they can fork a wikimedia labs project? :) I don't think anyone has every implemented this [16:32:21] throw the project on Github and let them fork it there? [16:32:25] or is there something I'm missing? [16:32:35] well, yes [16:32:47] setup of a tool may require more than just cloning a git repository [16:32:48] Nettrom: well, say you have a tool on labs that needs database access. It's useful to have some sort of fork as development copy [16:33:12] petan: in terms of bus factor: documenting those steps is also a good thing :-p [16:35:39] 3Wikimedia Labs / 3deployment-prep (beta): sync articles from production wikis (css/gadgets) - 10https://bugzilla.wikimedia.org/49779 (10Greg Grossmeier) a:5Ariel T. Glenn>3None [17:04:01] YuviPanda: yt? [18:19:23] 3Wikimedia Labs / 3tools: program created by proprietary compiler allowed on labs? - 10https://bugzilla.wikimedia.org/72253#c3 (10Luis Villa (WMF Legal)) As a legal matter, what I care about is that we, the Foundation, have permission to have the code on our servers without needing to review each individual... [18:49:42] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c2 (10Greg Grossmeier) Can we have timing information to help diagnose? See also: https://tools.wmflabs.org/nagf/?project=deployment-prep [18:49:57] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366 (10Greg Grossmeier) p:5Unprio>3High [18:55:09] !log upgraded hhvm on beta labs to 3.3.0+dfsg1-1+wm1 [18:55:10] upgraded is not a valid project. [18:55:16] !log deployment-prep upgraded hhvm on beta labs to 3.3.0+dfsg1-1+wm1 [18:55:20] Logged the message, Master [18:55:36] greg-g: pleaaaaase let's rename deployment-prep to beta-labs [18:57:59] ori: isn't that a pain in the ass? (also: omg yes) [18:58:13] so what? [18:58:22] * greg-g sighs [18:58:23] yes yes [18:58:42] andrewbogott: hey! I'm around now... [18:58:43] but I've been told by labsen that it's basically stupidly hard [18:58:55] I'll file a task [18:59:04] 'easiest' way is probably to re-create... [18:59:23] yeah [18:59:24] ori, greg-g, when you create the second beta cluster you should call /that/ one beta. [18:59:45] so yeah, not happening soon [18:59:47] Because don't you want one project that's actually beta and one that's actually, y'know, deployment-prep? [18:59:50] anyways, filing ticket [18:59:53] greg-g: beta cluster and alpha cluster :D [19:00:20] YuviPanda: I wanted to chat about auth for horizon. [19:00:27] andrewbogott: ah, sure! [19:00:28] The trivial case is to let it manage its own sessions [19:00:37] In which case it would not use 2fa [19:00:45] btw: we're delaying the second cluster idea until we have more people to help, right now we're too stretched thin. We will do multiversion (hetdeploy) on beta/-prep/whatever to have one doing what it does now (every 10 minutes updates) and then a nightyly that updates once/day [19:01:01] stretched too thin #grammar [19:01:14] andrewbogott: ah, hmm. wikitech will still be around, even if it isn't going to be used with OSM. Perhaps OAuth? [19:01:23] andrewbogott: we could also add OATH 2fa support to horizon [19:01:29] and just plug that into ldap [19:01:47] greg-g: how many more people are you looking at? I know you're hiring a release engineer currently. [19:01:54] JohnFLewis: yeah, that [19:02:02] that's all I have money for now :) [19:02:04] Yeah, I'm having a hard time thinking this through. It's already the case that when you authorize on wikitech you also get a keystone token. That token should, in theory, provide horizon access. [19:02:24] So one option is to do away with the horizon login screen althogether, and get horizon to just use the token from wikitech. [19:02:26] andrewbogott: is the keystone token public? [19:02:28] greg-g: hehe, enjoy the applicants :p [19:02:36] andrewbogott: ah, hmm. Although, making wikitech 'just a wiki' would be good [19:02:46] andrewbogott: since otherwise we'll have to deal with horizon, wikitech and the part that interacts.... [19:03:05] hm... [19:03:23] andrewbogott: so 'ideal' world it would plug into LDAP + implement 2FA [19:03:34] I haven't really thought through if we can move 100% of OSM functionality to Horizon. Certainly not right away. [19:03:42] But you're right, if the goal is to separate them entirely... [19:03:48] andrewbogott: why not, over time? [19:04:11] Well, we'll probably still want some auto-documentation to happen on wikitech. And currently those edits require a login session on wikitech. [19:04:14] 3Wikimedia Labs / 3deployment-prep (beta): Rename all occurences of "deployment-prep" to "beta-cluster" - 10https://bugzilla.wikimedia.org/72694 (10Greg Grossmeier) 3NEW p:3Unprio s:3minor a:3None "deployment-prep" no longer makes sense and it's not how anyone refers to the cluster of vms known as "B... [19:04:18] I guess all that could be moved to a bot account [19:04:28] JohnFLewis: :P [19:04:29] andrewbogott: why on wikitech? those things aren't human edited much. we should just have them be in horizon... [19:04:35] andrewbogott: and link to a wikitech page if needed from there [19:04:43] 3Wikimedia Labs / 3deployment-prep (beta): Rename all occurences of "deployment-prep" to "beta-cluster" - 10https://bugzilla.wikimedia.org/72694 (10Greg Grossmeier) p:5Unprio>3Low s:5minor>3enhanc [19:05:19] Well, pages like this are pretty handy https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack [19:05:22] greg-g: recreating the cluster sounds like a pain indeed.. [19:05:59] YuviPanda: so, by default Horizon uses keystone auth, which is backed by ldap. [19:06:06] JohnFLewis: no kidding :( [19:06:07] So if we separate the projects entirely, we already have almost exactly what we need. [19:06:31] And then I guess we can regard the 2fa question as entirely a keystone plugin issue. [19:06:38] Except it would be nice to share the factor with wikitech, huh? [19:07:10] andrewbogott: not necessarily. if we kill OSM stuff on wikitech, then 2FA isn't *as* important as it is now... [19:07:15] andrewbogott: so having them be seperate would be good [19:07:26] andrewbogott: in fact I think we *should* have them be seperate. security, etc [19:07:39] greg-g: it might be possible to move an instance from one project to another by fussing with the db. If you /really/ care about renaming, we could research that option. [19:07:56] Ah, right, so in the perfect future wikitech wouldn't use 2fa. [19:08:29] andrewbogott: well, wouldn't 'require' 2fa for everyone... [19:08:34] So -- sounds like this boils down to only one issue, which is: support 2fa in keystone. Do you agree? [19:08:41] yeah [19:08:57] But the 2nd factor would have to be in a db, not in ldap, huh? [19:09:07] Well, I guess maybe it could be in ldap, I guess the password is already [19:09:56] hm... [19:12:03] andrewbogott: sounds nasty but potentially easier? feel free to let the bug know what you think :) [19:12:22] greg-g: what /I/ think is: who cares what it's called? :) [19:12:32] :) [19:12:38] andrewbogott: I don't also know if 2FA is a 'blocker' to starting on Horizon [19:12:57] YuviPanda: it probably isn't, but there may well be existing implementations anyway [19:13:21] My concern is that right now the keystone/ldap model is read-only. It authenticates against ldap but can't create accounts or modify them. [19:13:26] andrewbogott: yeah, should be. we can't be the only ones [19:13:39] Which is fine if wikitech handles that. But by that logic, wikitech should also manage the second factor. [19:14:07] So I'm back to thinking that the ideal situation is having keystone just read the 2nd factor straight out of the oathauth db [19:14:12] which sounds like a custom job [19:15:21] um… oauthath [19:15:29] 3Wikimedia Labs / 3deployment-prep (beta): Rename all occurences of "deployment-prep" to "beta-cluster" - 10https://bugzilla.wikimedia.org/72694#c1 (10Andrew Bogott) It might be possible to create a new project and then yoink the instances out of deployment-prep and into the new one. That will break a fair... [19:16:23] andrewbogott: hmmm, maybe. but then you'll have to manage those tokens in wikitech... [19:16:39] andrewbogott: maybe we can make Wikitech's OATH code put the token in ldap? [19:17:11] Yeah, might be better. [19:17:49] I'll do some googling for keystone + 2fa. So far all I've found suggests that we just back keystone with oauth (which I guess would also be fine) [19:18:14] 3Wikimedia Labs / 3deployment-prep (beta): Rename all occurences of "deployment-prep" to "beta-cluster" - 10https://bugzilla.wikimedia.org/72694#c2 (10John F. Lewis) Hashar: Have any opinions regarding this? I spoke with Greg and offered to make good use of some of my spare time by basically doing the bulk o... [19:19:54] greg-g: The current name is confusing, but renaming seems like more work that it's really worth. I'd rather see us work on getting a new project setup, cleaning up the puppet code to use hiera, etc. [19:23:15] andrewbogott: yeah, that might be an intermediate step maybe... [19:23:26] andrewbogott: before moving everything over at some point in the future [19:25:13] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c3 (10Chris McMahon) example failure from just now: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/153/testReport/junit/(root)/... [19:25:19] bd808: fair point :) [19:31:05] bd808: btw, hiera for labs is almost here... [19:32:13] YuviPanda: I've been following along via znc buffer stalking. :) I've got some ideas of places that it will make beta easier to manage. [19:33:23] YuviPanda: This bug is one thing I think that hiera can mostly fix for us [19:52:13] andrewbogott: Are ip addresses for labs instances static? I'm seeing an error in beta where there is a config setting that uses an ip address and that ip is assigned to a different instance in a different project than the config expects. [19:52:38] andrewbogott: I'm wondering if this is a config typo or a bad assumption. [19:53:58] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c4 (10Greg Grossmeier) Reedy/Antoine: Can you two diagnose this one, please? Chris: Having more than two data points would be helpful (there might be a pattern eg tests running while code/... [19:57:42] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c5 (10Chris McMahon) Greg: it happens so often that I hadn't tried to track it closely. [20:01:30] bd808: are you talking about a public or private ip? [20:02:00] andrewbogott: private, the 10.68.x.x address [20:02:18] It's most likely a config typo, unless you know that it worked previously. [20:02:52] Looking at the comment, it may have been broken by rebuilding the instance. [20:03:15] The comment said deployment-redis1 and the right instance is -redis01 [20:03:25] So that is a likely cause [20:03:47] ok. There used to be a bug which caused occasional IP collisions. So if it's really happening then I'm interested, but it /should/ have been fixed months ago. [20:15:03] andrewbogott: I haven't seen that bug since [20:15:13] Me neither [20:36:41] 3Wikimedia Labs / 3deployment-prep (beta): Rename all occurences of "deployment-prep" to "beta-cluster" - 10https://bugzilla.wikimedia.org/72694#c3 (10Andre Klapper) (If you also want to rename the corresponding Bugzilla component, please file a ticket under "Wikimedia > Bugzilla") [20:58:00] 3Wikimedia Labs / 3deployment-prep (beta): Logs not being received by udp2log after PSR-3 patch - 10https://bugzilla.wikimedia.org/72701 (10Bryan Davis) 3NEW p:3Unprio s:3major a:3Bryan Davis After https://gerrit.wikimedia.org/r/#/c/119941/ landed I am not seeing logs in /data/project/logs or logstash. [20:58:28] 3Wikimedia Labs / 3deployment-prep (beta): Logs not being received by udp2log after PSR-3 patch - 10https://bugzilla.wikimedia.org/72701 (10Bryan Davis) p:5Unprio>3High [21:55:30] 3Tool Labs tools / 3wikibugs IRC bot: Newly filed bugs with empty description are not accounced on IRC - 10https://bugzilla.wikimedia.org/72708 (10Bartosz Dziewoński) 3NEW p:3Unprio s:3normal a:3Merlijn van Deen Newly filed bugs with empty description are not accounced on IRC. Bug 72705 wasn't pushed... [22:08:29] 3Tool Labs tools / 3wikibugs IRC bot: Newly filed bugs with empty description are not accounced on IRC - 10https://bugzilla.wikimedia.org/72708#c1 (10Merlijn van Deen) 5NEW>3RESO/DUP Yeah, Bugzilla doesn't send messages to wikibugs-l, so pywikibugs cannot report anything. See #64836. *** This bug has be... [22:23:42] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c6 (10Bryan Davis) 503s could very well could be caused by https://gerrit.wikimedia.org/r/#/c/163078/ which restarts the hhvm server as part of each scap. While the hhvm fcgi process is shu... [22:29:29] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c7 (10Greg Grossmeier) That is a really strong culprit.... [22:30:42] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c8 (10Ori Livneh) > Greg: it happens so often that I hadn't tried to track it closely. How often do we run scap? [22:31:42] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c9 (10Bryan Davis) (In reply to Ori Livneh from comment #8) > > Greg: it happens so often that I hadn't tried to track it closely. > > How often do we run scap? On every commit to core or... [22:41:57] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c10 (10Chris McMahon) 163078 was merged 14 October, the timing looks good for this being the culprit. It seems to occur with increasing frequency, but that could be my bias. [22:45:42] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c11 (10Bryan Davis) (In reply to Chris McMahon from comment #10) > 163078 was merged 14 October, the timing looks good for this being the > culprit. > > It seems to occur with increasing f... [22:46:13] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c12 (10Greg Grossmeier) (In reply to Bryan Davis from comment #9) > (In reply to Ori Livneh from comment #8) > > > Greg: it happens so often that I hadn't tried to track it closely. > > >... [22:47:27] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c13 (10Greg Grossmeier) (In reply to Bryan Davis from comment #11) > (In reply to Chris McMahon from comment #10) > > 163078 was merged 14 October, the timing looks good for this being the... [23:04:41] 3Wikimedia Labs / 3deployment-prep (beta): Logs not being received by udp2log after PSR-3 patch - 10https://bugzilla.wikimedia.org/72701#c1 (10Bryan Davis) 5NEW>3RESO/FIX This turned out to be caused by an old and known problem. The udp2log service was running on deployment-bastion instead of the needed... [23:04:45] 3Wikimedia Labs / 3deployment-prep (beta): [OPS] udp2log prevents udp2log-mw from starting - 10https://bugzilla.wikimedia.org/38995 (10Bryan Davis) [23:09:59] 3Wikimedia Labs / 3deployment-prep (beta): HHVM fcgi restart during scap runs cause 503s (and failed tests) - 10https://bugzilla.wikimedia.org/72366 (10Greg Grossmeier) s:5normal>3major [23:10:15] 3Wikimedia Labs / 3deployment-prep (beta): HHVM fcgi restart during scap runs cause 503s (and failed tests) - 10https://bugzilla.wikimedia.org/72366#c16 (10Greg Grossmeier) (In reply to Bryan Davis from comment #14) > Giuseppe is interested in adding etcd support to pybal that could make this > possible but... [23:10:29] 3Wikimedia Labs / 3deployment-prep (beta): HHVM fcgi restart during scap runs cause 503s (and failed tests) - 10https://bugzilla.wikimedia.org/72366#c14 (10Bryan Davis) (In reply to Greg Grossmeier from comment #13) > (In reply to Bryan Davis from comment #11) > > > > If restarting hhvm is the cause, the fr... [23:11:28] 3Wikimedia Labs / 3deployment-prep (beta): HHVM fcgi restart during scap runs cause 503s (and failed tests) - 10https://bugzilla.wikimedia.org/72366 (10Greg Grossmeier) [23:11:58] 3Wikimedia Labs / 3deployment-prep (beta): HHVM fcgi restart during scap runs cause 503s (and failed tests) - 10https://bugzilla.wikimedia.org/72366#c15 (10Chris McMahon) "a bit"? [23:14:39] hey, I go to https://wikitech.wikimedia.org/wiki/Special:NovaInstance and add some project filters (deployment-prep, editor-engagement) and [Set filter]... and I don't see any instances. I think this used to work. [23:16:46] spagewmf: are you a projectadmin of those instances? [23:17:00] spagewmf: log out, log back in. [23:17:01] 3Wikimedia Labs / 3deployment-prep (beta): HHVM fcgi restart during scap runs cause 503s (and failed tests) - 10https://bugzilla.wikimedia.org/72366#c17 (10Ori Livneh) I think someone should actually do the legwork to investigate this issue by cross-referencing 503s against the log files before we get carrie... [23:17:33] JohnFLewis: yes for editor-engagmement. [23:18:04] My workaround is go to https://wikitech.wikimedia.org/wiki/Nova_Resource:Editor-engagement which shows "Instances for this project" [23:19:51] my actual issue is http://ee-dashboard.wmflabs.org/dashboards/enwiki-metrics seems like a labs server that should be in project editor-engagement, but isn't. I was trying to track down its project [23:38:44] 3Wikimedia Labs / 3deployment-prep (beta): HHVM fcgi restart during scap runs cause 503s (and failed tests) - 10https://bugzilla.wikimedia.org/72366#c18 (10Greg Grossmeier) (In reply to Ori Livneh from comment #17) > I think someone should actually do the legwork to investigate this issue by > cross-referenc...