[06:40:59] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 531 bytes in 0.014 second response time
[06:56:03] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.029 second response time
[10:33:41] <wikibugs>	 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2775777 (10jcrespo) @russblau Thanks for the report- it is 5 as we speak, but it is indeed wrong. This and hopefully all drift issues are fixed on the imports on the new labsdb servers T147052, that I hope they will be soon...
[10:41:59] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2775785 (10jcrespo) @russblau See my comment at T138967#2775777. Expanding on that, import "in place" created lots of disruption (replication lag, which other users complained about). The decision take...
[12:01:56] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Morteza was created, changed by Morteza link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Morteza edit summary: Created page with "{{Tools Access Request |Justification=I want to create robots, tools and some accessories. |Completed=false |User Name=Morteza }}"
[13:32:15] <wikibugs>	 10Tool-Labs-tools-Other, 06translatewiki.net, 07I18n, 13Patch-For-Review: [[Intuition:Monumentsapi-title/en]] i18n issue - https://phabricator.wikimedia.org/T137951#2776135 (10Nemo_bis) 05Open>03Resolved a:03Nemo_bis
[14:18:03] <wikibugs>	 10Tool-Labs-tools-Other, 10Wiki-Loves-Monuments, 06translatewiki.net, 07I18n, 13Patch-For-Review: [[Intuition:Monumentsapi-title/en]] i18n issue - https://phabricator.wikimedia.org/T137951#2776232 (10Lokal_Profil)
[15:32:01] <wikibugs>	 10PAWS: paws.wmflabs.org should be the public view - https://phabricator.wikimedia.org/T150180#2776504 (10Halfak)
[15:32:54] <wikibugs>	 10PAWS: paws.wmflabs.org should be the public view - https://phabricator.wikimedia.org/T150180#2776504 (10Halfak)
[15:52:52] <wikibugs>	 10Labs-project-other, 06Developer-Relations: move WikiApiary to Labs - https://phabricator.wikimedia.org/T149874#2767311 (10Andrew) This is probably fine, but can you please provide a summary of what this project is, what it will do, and why it needs a project and can't be a tools webservice?  thanks!
[16:15:23] <grrrit-wm>	 (03Draft1) 10Paladox: grrrit-wm: Improve logging [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 
[16:15:26] <grrrit-wm>	 (03Draft2) 10Paladox: grrrit-wm: Improve logging [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 
[16:15:48] <grrrit-wm>	 (03CR) 10Paladox: "I'm not entirely sure this is reliable." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox)
[16:23:01] <grrrit-wm>	 (03CR) 10Paladox: "recheck" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox)
[16:44:52] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 07Availability: Decide between proxysql and haproxy for labsdbproxy service - https://phabricator.wikimedia.org/T149844#2776848 (10jcrespo)
[17:16:00] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, and 2 others: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2776977 (10mark) Approved.
[17:24:23] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, and 2 others: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2777041 (10jcrespo) a:05jcrespo>03None @RobH You mentioned it may not need a physica...
[17:28:03] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, and 2 others: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2777052 (10RobH) I said that in reply to the labs to db server transition, not the trans...
[17:30:10] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, and 2 others: Move dbproxy1010 and dbproxy1011 to labs-support network, rename them to labsdbproxy1001 and labsdbproxy1002 - https://phabricator.wikimedia.org/T149170#2777055 (10jcrespo) Sorry for the missunderstanding!
[17:38:21] <wikibugs>	 10PAWS: paws.wmflabs.org should be the public view - https://phabricator.wikimedia.org/T150180#2776504 (10Capt_Swing) I agree with this logic, and shall therefore award this task some sort of amusing token.
[17:45:10] <wikibugs>	 10PAWS: Re-render index from a Wiki page - https://phabricator.wikimedia.org/T150131#2774853 (10Capt_Swing) So, the idea is that anyone can add public links to any interesting Jupyter notebooks or directories //of which they are aware// to the MediaWiki page, even if they are someone else's notebooks? That makes...
[18:01:08] <andrewbogott>	 aude: property-suggester.wikdata-dev.eqiad.wmflabs seems sort of broken… is that instance still in current use?
[18:25:08] <DatGuy>	 anyone here knows how to use MySQL db? More specifically, I am getting a "access denied" OperationalError
[18:34:33] <bd808>	 DatGuy: that sounds like you are not using a username and password or are using a username and password that aren't allowed for the particular database you are accessing. Can you give us some additional context about what you are trying to do?
[18:35:19] <DatGuy>	 So, I'm trying to run a bot. Here is the .err issue:
[18:35:42] <DatGuy>	 http://pastebin.com/8Ffjr9K9
[18:36:42] <wikibugs>	 10Labs-project-other: Configure Single Sign On at discourse.wmflabs.org - https://phabricator.wikimedia.org/T124691#2777284 (10jayvdb) What about using [[https://secure.phabricator.com/book/phabcontrib/article/using_oauthserver/ | Phabricator OAuth server]] as a temporary workaround.  It may be a bit of fiddle t...
[18:37:19] <DatGuy>	 http://pastebin.com/mWbNjZUW is the full error
[18:37:54] <Krenair>	 'datguy', using password: NO?
[18:38:01] <Krenair>	 yeah your read_default_file isn't working
[18:38:06] <bd808>	 "Access denied for user 'datguy'@'10.68.23.223' (using password: NO)" -- so read_default_file isn't working
[18:39:40] <bd808>	 legoktm has some nice little helper for using the conf file. I think it's on wikitech somewhere...
[18:39:59] <bd808>	 https://wikitech.wikimedia.org/wiki/User:Legoktm/wmflib
[18:40:48] <bd808>	 hmmm... it also seems to use read_default_file -- https://github.com/legoktm/wmflabs-lib/blob/master/wmflabs/db.py#L26
[18:41:28] <Krenair>	 I use this:
[18:41:33] <Krenair>	 from ConfigParser import ConfigParser
[18:41:33] <Krenair>	 config = ConfigParser()
[18:41:33] <Krenair>	 config.read('replica.my.cnf')
[18:41:34] <Krenair>	 user = config.get('client', 'user')[1:-1] # Strip first and last characters - just apostrophes
[18:41:34] <Krenair>	 password = config.get('client', 'password')[1:-1] # Strip first and last characters - just apostrophes
[18:43:00] <bd808>	 DatGuy: did you edit your replica.my.cnf file? Maybe the whitespace and lack of quotes is breaking the parser?
[18:43:18] <DatGuy>	 never edited it, don't even have permission
[18:43:24] <DatGuy>	 checking for whitespace
[18:43:27] <Krenair>	 woah, yeah, that doesn't look like a normal one
[18:43:48] <Krenair>	 my code won't work for that
[18:45:06] <bd808>	 Krenair: hmm... looks like that's how new tools are getting their files created
[18:45:12] <Krenair>	 ugh
[18:45:22] <bd808>	 the one for my-first-django-tool looks like that too
[18:45:27] <DatGuy>	 Is it fine if I post 3-line code here, or should I pbin it too?
[18:45:47] <bd808>	 DatGuy: don't post or pastebin
[18:45:54] <bd808>	 Krenair and I can see it with root powers
[18:46:15] <bd808>	 DatGuy: let me edit it so it looks "normal" and then you can try again
[18:46:23] <DatGuy>	 alright
[18:48:25] <bd808>	 wtf. root can't write it
[18:48:46] <DatGuy>	 hm
[18:48:47] <Krenair>	 something to do with nfs?
[18:49:39] <Krenair>	 seems this affects approximately 439 tools
[18:50:18] <bd808>	 as root: chmod: changing permissions of ‘replica.my.cnf’: Operation not permit
[18:50:24] <bd808>	 that's "fun"
[18:50:52] <DatGuy>	 is it possible for me to change it?
[18:51:28] <wikibugs>	 06Labs, 07Puppet: Puppet parser, puppet API, and inline docs - https://phabricator.wikimedia.org/T148479#2777347 (10Andrew)
[18:51:31] <wikibugs>	 06Labs: Move all of Labs (including self-hosted masters) to puppet 3.8.5 - https://phabricator.wikimedia.org/T148431#2777346 (10Andrew) 05Open>03Resolved
[18:52:06] <bd808>	 DatGuy: apparently not. I tried as the tool user too.
[18:52:08] <Krenair>	 of like 1576 tools
[18:54:26] <bd808>	 DatGuy: I made a /data/project/datbot/test.my.cnf file. See if that one works for you
[18:54:46] <bd808>	 This may or may not be the problem, but we should rule it out at least
[18:56:49] <DatGuy>	 submitted..
[18:57:45] <DatGuy>	 (1045, "Access denied for user 'datguy'@'10.68.18.17' (using password: NO)")
[18:58:25] <bd808>	 ok, so that is ruled out. The problem is somehow that the file contents are not being used by mysql and not the specific formatting
[18:58:55] <DatGuy>	 stopped the job
[19:00:06] <Krenair>	 I don't htink that parameter you're using works DatGuy 
[19:00:11] <Krenair>	 >>> MySQLdb.connect(host = "s7.labsdb", db = "meta_p", read_default_file = 'replica.my.cnf')
[19:00:14] <Krenair>	 _mysql_exceptions.OperationalError: (1045, "Access denied for user 'krenair'@'10.68.23.58' (using password: NO)")
[19:00:18] <bd808>	 DatGuy: legoktm's code uses the https://pythonhosted.org/oursql/ library instead of MySQLdb. 
[19:00:30] <Krenair>	 this is on my personal user with credentials from March 2014
[19:00:49] <DatGuy>	 Mr.Z-bot uses similar code I believe
[19:02:25] <bd808>	 from the docs at http://mysql-python.sourceforge.net/MySQLdb.html it looks like read_default_file may only effect mysql_options() things
[19:03:08] <bd808>	 so you may need to use something like Krenair posted earlier where you read the username + password yourself or switch to oursql
[19:04:53] <DatGuy>	 if I do either, it won't force me to change quite a bit of the code?
[19:05:05] <Krenair>	 you probably only do a DB connection in one place
[19:05:21] <Krenair>	 it's like 5 lines
[19:05:29] <DatGuy>	 alright, that's nice
[19:05:33] <bd808>	 reading the user+password would probably be the least disruptive change
[19:08:50] <DatGuy>	 If I am using Krenair's suggestion, then what should I place at "client", "user", and "password". As far as I know, user is just datguy, and I don't have a password
[19:09:18] <Krenair>	 no
[19:09:21] <Krenair>	 user is not datguy
[19:09:49] <Krenair>	 your credentials for mysql are found in your replica.my.cnf file
[19:10:22] <DatGuy>	 ah yes. My personal ones, correct? Not the datbot project one.
[19:10:52] <bd808>	 no, actually "your" here means your tool's data
[19:11:11] <bd808>	 so the data from /data/project/datbot/replica.my.cnf
[19:11:19] <DatGuy>	 Don't have access to that.
[19:11:30] <bd808>	 you have to become  the tool
[19:11:40] <DatGuy>	 sigh, I'm using WinSCP
[19:12:05] <DatGuy>	 I'll tell you when its changed and if it worked
[19:12:14] <DatGuy>	 thanks guys
[19:12:18] <bd808>	 ah. you are editing from windows and scp'ing the data up
[19:12:52] <DatGuy>	 Yep. sadly, I've never used linux. It's quite good. I primarily use Windows because of more application options
[19:13:48] <bd808>	 The code that Krenair gave doesn't really care what the contents of the file are. The only change I would make from his example is to replace the explict slicing of the string with a strip() that drops single quote (') and whitespace instead
[19:13:54] <DatGuy>	 client just stays, correct?
[19:14:37] <bd808>	 let me test and then make a paste for you to look at
[19:24:13] <bd808>	 This seems to work from one of my test tools -- https://phabricator.wikimedia.org/P4383
[19:25:30] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2777555 (10leila)
[19:25:39] <DatGuy>	 and I can put that anywhere, correct? Not just on top
[19:26:11] <bd808>	 yeah. it could be in a function or method or where ever it's needed
[19:26:21] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2689440 (10leila) @bd808 I'm removing Research-and-Data, but I myself will stay on this task. Let's discuss the analysis whenever you're ready. :)
[19:27:09] <DatGuy>	 and final thing is that the other commands (db.autocommit(True) and cursor = db.cursor()) stay?
[19:27:27] <bd808>	 sure. those would be fine
[19:28:26] <bd808>	 reading the username and password from the replica.my.cnf file and then using them explicitly in creating the db object is the important part that was failing
[19:28:28] <DatGuy>	 do you still have afreporter.py open? If so, see line 149. That should also be changed?
[19:30:21] <bd808>	 I actually never looked at that file. I can. Hang on a minute
[19:30:54] <DatGuy>	 http://pastebin.com/vFztPfep is what I changed it do
[19:30:55] <DatGuy>	 to*
[19:31:29] <DatGuy>	 previously it was http://pastebin.com/GrJwQjSQ
[19:32:07] <bd808>	 DatGuy: that looks right to me
[19:32:50] <DatGuy>	 by the way, I'm using jstart -mem 2G -N afreporter /data/project/datbot/bot2/bin/python /data/project/datbot/Filter-reporter/afreporter.py to start
[19:34:01] <DatGuy>	 there are now other errors, but I believe I can debug them on my own :). Thanks.
[19:34:10] <bd808>	 DatGuy: yw
[19:34:21] <bd808>	 help the next person you see ask that question :)
[19:34:34] * DatGuy keeps note
[19:54:04] <DatGuy>	 Hello again guys :P. I've fixed two issues, but now one that I can't figure out popped up. http://pastebin.com/QhjD1SbR is the error, and http://pastebin.com/rmAqaVH1 is the specific code
[19:56:09] <Platonides>	 your variable user is a string
[19:56:22] <Platonides>	 and you seem to expect a module
[19:56:32] <wikibugs>	 10Labs-project-other, 06Developer-Relations: move WikiApiary to Labs - https://phabricator.wikimedia.org/T149874#2777714 (10Dzahn) what it is and will do:  "WikiApiary collects, displays and analyzes information about MediaWiki websites. Once a website is registered with WikiApiary a suite of bots will start c...
[19:57:09] <DatGuy>	 I believe it is Wikitools (a library)
[19:57:13] <DatGuy>	 also, why does it work for the others?
[19:59:42] <Platonides>	 the error is outside your excerpt
[19:59:50] <Platonides>	 look for a user = …  line
[20:00:29] <DatGuy>	 I have only user = for a SQL thing that you can see above
[20:00:49] <DatGuy>	 should I pastebin all of main()?
[20:07:24] <DatGuy>	 Platonides all of main = http://pastebin.com/tLye27Ln
[20:09:24] <Platonides>	 do you have an import user  or similar?
[20:10:37] <DatGuy>	 http://pastebin.com/tZW8s1Yb
[20:10:50] <DatGuy>	 Not really, I believe you are referring to wikitools?
[20:11:22] <DatGuy>	 There is in wikitools:
[20:11:38] <DatGuy>	 http://pastebin.com/VycskFaf
[20:13:21] <Platonides>	 how is that file named?
[20:13:48] <DatGuy>	 user.py, inside a directory called wikitools
[20:14:37] <Platonides>	 and there was presumible an from wikitools import *  in your main file
[20:15:16] <DatGuy>	 yes
[20:16:00] <Platonides>	 ok, that's the user you were expecting
[20:16:22] <Platonides>	 the odd thing is that instead of that user, you get a different one…
[20:16:47] <DatGuy>	 it expects it as a string
[20:16:59] <Platonides>	 are you sure you're not assigning anything to a variable named user?
[20:17:05] <Platonides>	 no
[20:17:08] <DatGuy>	 entry['u'] = row['user']
[20:17:35] <Platonides>	 user = 
[20:18:00] <DatGuy>	 Yes
[20:18:01] <DatGuy>	 I am
[20:18:48] <Platonides>	 *something* is overwriting user
[20:19:02] <DatGuy>	 yeah, I have a variableas user aside from that
[20:19:18] <DatGuy>	 not sure if it can be changed though
[20:19:31] <Platonides>	 why not?
[20:20:04] <DatGuy>	 Its related to SQL. I'll try to change it to user1
[20:20:39] <Platonides>	 that should fix it
[20:21:55] <DatGuy>	 TypeError: 'user1' is an invalid keyword argument for this function
[20:23:00] <Platonides>	 user=user1
[20:24:50] <DatGuy>	 done that
[20:24:54] <DatGuy>	 going to sleep, hopefully it works
[20:24:59] <Platonides>	 does it work now?
[20:25:00] <shinken-wm>	 PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[20:28:26] <DatGuy>	 probably
[20:28:29] <DatGuy>	 no .err file popped up yet
[20:29:03] <DatGuy>	 don't think its started yet actually
[20:29:20] <DatGuy>	 might be theres a problem? Is that was the pupopet says?
[20:30:16] <Platonides>	 no, I'm sure the message by shinken-wm is completely unrelated to your changes :)
[20:33:51] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2777843 (10bd808) @leila Do you have time to do the initial pass at making a normalized set from the survey data like the spreadsheet that existed when I got involved in the...
[20:45:06] <halfak>	 !log ores restarted uwsgi-ores and precached on ores-web-03
[20:45:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[20:47:30] <StonedCraft16>	 Hello
[20:57:17] <Krinkle>	 andrewbogott: I'd like to the big cvn nodes to slightly smaller ones. Main objective is also to get PHP5.5 and Trusty instead of Precise/5.3. Only issue is that in order to do so I need an extra public IP since the bots require one to avoid freenode limits. I have a new node set up but starting bots keeps failing due to the limit. 
[20:58:13] <Krinkle>	 cvn-app4 (xlarge) and cvn-app5 (xlarge) to go. and cvn-app6 (large) to be the migration target. Will create cvn-app7 (large) as well once one of them is decom'ed. 
[20:59:11] <andrewbogott>	 Krinkle: for an extra IP file a subtask here:  https://phabricator.wikimedia.org/T140904
[20:59:16] <andrewbogott>	 Krinkle: trusty and not jessie?
[20:59:32] <Krinkle>	 andrewbogott: Not yet. The web server is jessie now though (cvn-apache*)
[21:01:47] <wikibugs>	 06Labs: Request increased quota (floating ip) for "cvn" labs project - https://phabricator.wikimedia.org/T150209#2777950 (10Krinkle)
[21:01:51] <Krinkle>	 andrewbogott: ^
[21:02:08] <andrewbogott>	 Krinkle: thanks
[21:03:10] <Krinkle>	 andrewbogott: The cvnbots are more CPU bound than anything. So if a custom flavour or some other option that pre-exxists but not exposed for cvn yet, that would be interesting as well to save resoures.
[21:03:27] <Krinkle>	 Here's the CPU graph for 1 of 2 main app nodes: https://tools.wmflabs.org/nagf/?project=cvn#h_cvn-app5_cpu
[21:03:29] <Krinkle>	 That is an xlarge
[21:04:18] <Krinkle>	 2-large is probably enough still. But anything smaller and CPU becomes an issue. But all other resources are practically idle and mostly unused.
[21:04:24] <Krinkle>	 much smaller disk/ram would be fine.
[21:04:26] <Krinkle>	 Up to you :)
[21:04:41] <shinken-wm>	 RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:04:56] <andrewbogott>	 ok — large instances are probably fine but I'll think about it
[21:09:14] <andrewbogott>	 bd808: any idea how to modify bot rights on wikitech?  My bot seems able to create/edit pages but not delete them
[21:09:27] <andrewbogott>	 (which is a sensible default, but I want to clean up after myself)
[21:09:44] <StonedCraft16>	 How long will the phabricator.wikimedia.org be down for maintenance
[21:10:02] <wikibugs>	 06Labs: Request increased quota (floating ip) for "cvn" labs project - https://phabricator.wikimedia.org/T150209#2777997 (10Krinkle)
[21:10:10] <andrewbogott>	 StonedCraft16: works for me...
[21:10:28] <StonedCraft16>	 yea iknow i refreshed just now and it worked
[21:10:34] <StonedCraft16>	 i know*
[21:10:43] <bd808>	 andrewbogott: what's the wikitech account?
[21:11:20] <andrewbogott>	 bd808: labslogbot
[21:11:38] <myrcx>	 Could someone murder my inefficient long running quarry query if/when they have a mo? - https://quarry.wmflabs.org/query/13928
[21:12:25] <bd808>	 andrewbogott: the generic answer is to go to https://wikitech.wikimedia.org/w/index.php?title=Special%3AUserRights&user=Labslogbot and grant the account membership in a group that gives it delete rights.
[21:12:39] <bd808>	 andrewbogott: I'll do that for you if you'd like
[21:13:09] <andrewbogott>	 bd808: do any of those checkboxes do that without conferring other super dangerous rights?
[21:13:29] <andrewbogott>	 I guess 'confirmed user' is probably pretty safe
[21:14:03] <bd808>	 from https://wikitech.wikimedia.org/wiki/Special:ListGroupRights it looks like you need either contentadmin or admin to get the delete right. contentadmin is the less scary of those two
[21:14:16] <bd808>	 or we could create a new group that just has delete
[21:15:08] <andrewbogott>	 or I could just abandon https://phabricator.wikimedia.org/T140298
[21:15:25] <andrewbogott>	 But new group sounds like the safest option
[21:15:49] <bd808>	 andrewbogott: contentadmin looks ok. It's got a lot of rights but none are too scary
[21:15:57] <andrewbogott>	 ok
[21:16:00] <andrewbogott>	 I'll try, thanks
[21:16:14] <bd808>	 its basically the "sysop minus ability to edit the UI js pages" role
[21:17:03] <bd808>	 Now I curious waht groups have delete on a more "normal" wiki
[21:17:48] <bd808>	 huh. on enwiki you can only get it from sysop
[21:19:10] <bd808>	 looks like that's normal -- delete is sysop only generally.
[21:19:29] <bd808>	 which kind of makes sense. deleting is not super common
[21:19:51] <andrewbogott>	 not super common and potentially very destructive
[21:19:54] <bd808>	 andrewbogott: autoconfirmed can move...
[21:20:15] <bd808>	 so it could move things to another namespace I guess
[21:21:10] <bd808>	 namespace 110 is "Obsolete" so you could move there I suppose
[21:21:36] <bd808>	 I'm not sure the old instance pages really have a lot of value though
[21:21:50] <andrewbogott>	 Yeah, they're not useful and we churn through a lot of them
[21:21:57] <bd808>	 so +1 for contentadmin and actual deleting
[21:21:58] <andrewbogott>	 which makes backups expensive and pollutes reporting and such
[21:22:06] <andrewbogott>	 ok
[21:22:20] <andrewbogott>	 seems to work with that additional right
[21:22:21] <andrewbogott>	 https://wikitech.wikimedia.org/wiki/Nova_Resource:Statuspage05.testlabs.eqiad.wmflabs
[21:22:33] <bd808>	 I'm not sure that "delete" makes anything less heavy for backups
[21:22:59] <bd808>	 deleted pages aren't really deleted, just move to other tables
[21:23:13] <bd808>	 hence the (view/restore) links
[21:23:43] <andrewbogott>	 I think that we only upload the current versions to wikitechstatic, not history
[21:23:55] <bd808>	 ah. that would make sense
[21:23:58] <andrewbogott>	 if that's right then a deleted page is probably better…  unless it copies up the 'this page has been deleted' record
[21:24:10] <andrewbogott>	 which is just as big as the not-very-big page would have been anyway
[21:26:12] <wikibugs>	 06Labs, 10wikitech.wikimedia.org: Labs front-page statistics are very wrong - https://phabricator.wikimedia.org/T139773#2778064 (10Andrew)
[21:26:14] <wikibugs>	 06Labs, 13Patch-For-Review: Actually delete instance status pages when the instance is destroyed - https://phabricator.wikimedia.org/T140298#2778063 (10Andrew) 05Open>03Resolved
[21:49:12] <grrrit-wm>	 (03PS1) 10Paladox: Replacing swig with swig-templates [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320294 
[21:52:40] <ottomata>	 hi yallls, is there some documentation somewhere about how to use puppet in labs now?
[21:52:45] <ottomata>	 i'm trying to figrure out horizon...
[21:55:13] <Krenair>	 should be pretty simple
[21:55:19] <Krenair>	 find instance
[21:55:23] <Krenair>	 click puppet configuration
[21:55:27] <Krenair>	 modify roles/hiera data
[21:55:31] <ottomata>	 i just tried to do apply a standalone puppet master thing
[21:55:35] <ottomata>	 applied the role
[21:55:38] <ottomata>	 nothing happening though
[21:55:41] <Krenair>	 alternatively via projects or prefixes
[21:55:42] <Krenair>	 hm
[21:55:46] <Krenair>	 what instance?
[21:55:47] <ottomata>	 also edited hiera for this node to point puppetmaster
[21:55:53] <ottomata>	 kafka601.analytics
[21:56:03] <ottomata>	 i also coudln't figure out how to add new roles
[21:56:04] <ottomata>	 to the interface
[21:57:00] <Krenair>	 krenair@tools-bastion-03:~$ curl labcontrol1001.wikimedia.org:8100/v1/kafka601/node/kafka601.analytics.eqiad.wmflabs
[21:57:00] <Krenair>	 hiera: {}
[21:57:01] <Krenair>	 roles: []
[21:57:04] <Krenair>	 oh, wait
[21:57:28] <Krenair>	 krenair@tools-bastion-03:~$ curl labcontrol1001.wikimedia.org:8100/v1/analytics/node/kafka601.analytics.eqiad.wmflabs
[21:57:29] <Krenair>	 hiera: {puppetmaster: kafka601.analytics.eqiad.wmflabs}
[21:57:29] <Krenair>	 roles: ['role::kafka::analytics::broker', 'role::puppet::self', 'role::puppetmaster::standalone']
[21:57:43] <ottomata>	 yeah, those must be leftover from before the migration to horizon
[21:57:47] <ottomata>	 dunno where those are applied from now
[21:57:52] <ottomata>	 i can't see them in the ui
[21:58:06] <ottomata>	 ohhh, but if role::puppet::self was set
[21:58:13] <ottomata>	 i betcah its pointing at a different node of mine for puppetmaster
[21:58:22] <ottomata>	 which i think i know which one
[21:58:26] <Krenair>	 can you put me in the project?
[21:58:30] <ottomata>	 and that is good enough for me to work...but i dunno how i'd use the interface
[21:58:30] <ottomata>	 sure
[21:58:40] <Krenair>	 as a projectadmin
[22:00:56] <ottomata>	 Krenair:  think i just did
[22:01:36] <Krenair>	 yep
[22:02:26] <Krenair>	 ottomata, okay, so
[22:02:40] <Krenair>	 kafka::analytics::broker, puppet::self and puppetmaster::standalone are all listed as applied to this node in horizon -
[22:02:50] <Krenair>	 you just need to click the 'all (449)' tab to see it
[22:02:58] <Krenair>	 instead of 'common (5)'
[22:03:05] <ottomata>	 OH
[22:03:07] <ottomata>	 huh.
[22:03:53] <ottomata>	 Ah, ! ok perfect
[22:04:15] <ottomata>	 Krenair:  i suppose this list of roles is read from somewhere other than this nodes puppetmaster?
[22:04:31] <ottomata>	 if i were to add a role to the standalone puppetmaster's manifests, it wouldn't show up here, right?
[22:04:36] <Krenair>	 yeah the lists of roles and hieradata are always kept on the central labs puppetmaster
[22:04:44] <ottomata>	 aye
[22:04:54] <Krenair>	 you can add extra classes not listed there using the 'classes' hiera key
[22:05:09] <ottomata>	 ah ok cool
[22:05:16] <ottomata>	 that includes them on the node
[22:05:18] <ottomata>	 ?
[22:05:20] <ottomata>	 just by doing
[22:05:20] <Krenair>	 puppetmasters query the central labs puppetmaster to determine which roles to apply
[22:05:21] <Krenair>	 yes
[22:05:22] <ottomata>	 classes:
[22:05:26] <ottomata>	 - my::class
[22:05:26] <ottomata>	 ?
[22:05:30] <ottomata>	 cool
[22:05:31] <ottomata>	 thanks
[22:05:42] <Krenair>	 though obviously you need to use the role:: prefix on them there
[22:06:04] <Krenair>	 so the other thing was... it didn't seem to actually apply the change?
[22:06:45] <ottomata>	 which chnage?
[22:06:55] <ottomata>	 (I removed puppet standalone, because the old puppet self was applied
[22:06:55] <Krenair>	 your addition of puppetmaster::standalone
[22:06:58] <ottomata>	 and still on a working node
[22:07:01] <Krenair>	 ah
[22:07:05] <ottomata>	 so i'm just tryign to test some puppet stuff for now
[22:07:08] <ottomata>	 and since that seems to be working
[22:07:12] <ottomata>	 i've fixed up that old puppet master
[22:07:18] <ottomata>	 and will just try stuff from there
[22:07:21] <Krenair>	 ok
[22:07:23] <ottomata>	 hmmm
[22:07:24] <ottomata>	 Error: Could not retrieve catalog from remote server: Error 400 on SERVER: pick_initscript(): Wrong number of arguments given (7 for 6) at /etc/puppet/modules/base/manifests/service_unit.pp:90 on node kafka601.analytics.eqiad.wmflabs
[22:07:26] <ottomata>	 sigh
[22:21:41] <ottomata>	 Krenair:  ever seen this before?
[22:21:41] <ottomata>	 Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://kafka601.analytics.eqiad.wmflabs/plugins: Connection refused - connect(2) for "kafka601.analytics.eqiad.wmflabs" port 8140
[22:21:45] <ottomata>	 it was working a sec ago
[22:21:47] <ottomata>	 now its not :/
[22:21:55] <ottomata>	 maybe i should just start with a new node...yargh
[22:22:06] <ottomata>	 Warning: Unable to fetch my node definition, but the agent run will continue:
[22:22:07] <ottomata>	 Warning: Connection refused - connect(2) for "kafka601.analytics.eqiad.wmflabs" port 8140
[22:25:11] <Krenair>	 ottomata, you tried to make it it's own puppetmaster
[22:25:16] <Krenair>	 but it doesn't run a puppetmaster service
[22:25:23] <Krenair>	 doesn't seem to have it installed
[22:25:27] <Krenair>	 let alone running
[22:25:37] <ottomata>	 HMMM
[22:25:41] <ottomata>	 its puppetmaster should be kafka401
[22:25:43] <ottomata>	 not 601
[22:25:51] <ottomata>	 but, that does give me a clue...
[22:26:02] <ottomata>	 seems good though:
[22:26:02] <ottomata>	 server = kafka401.analytics.eqiad.wmflabs
[22:26:07] <ottomata>	 [master]
[22:26:07] <ottomata>	 bindaddress = 10.68.16.152
[22:26:07] <ottomata>	 certname = kafka401.analytics.eqiad.wmflabs
[22:26:29] <ottomata>	 oh
[22:26:31] <ottomata>	 sorry, wrong noe..
[22:26:35] <Krenair>	 was about to say
[22:26:38] <ottomata>	 YEAH
[22:26:39] <Krenair>	 that's not the file I'm looking at :)
[22:26:40] <ottomata>	 what changed it?!
[22:26:57] <Krenair>	 ?
[22:27:09] <ottomata>	 hmm, i wonder what happened to the wikitech puppet variables this thing used to use
[22:27:16] <ottomata>	 that's how we assigned puppetmaster in role puppet self before
[22:27:31] <Krenair>	 we wrote the horizon panel to replace that
[22:27:41] <ottomata>	 hmm, oh and i edied this hiera
[22:27:42] <ottomata>	 ok hm
[22:27:50] <ottomata>	 maybe it overrides..?  hm
[22:29:15] <Krenair>	 what do you think overrides what?
[22:29:55] <ottomata>	 i set puppetmaster in horizon hiera for this node whne i was following instructions for standalone puppetmaster, before i knew that the old role puppet self was still applied
[22:29:59] <ottomata>	 just set it to 401
[22:30:00] <ottomata>	 we will see...
[22:30:22] <Krenair>	 I still haven't figured out in which direction you're trying to take this, but
[22:30:23] <Krenair>	 https://wikitech.wikimedia.org/wiki/Hiera:Analytics/host/kafka601
[22:30:42] <Krenair>	 puppet::self isn't going to set up a puppetmaster service if you set ::master to another host
[22:31:05] <ottomata>	 right, it used to be able to point to a remote puppetmaster
[22:31:10] <ottomata>	 which i already have running on 401
[22:31:32] <ottomata>	 all i want to do is run puppet and test some stuff!  i already had a self hosted puppetmaster for several of these nodes, so i'm trying to do that
[22:31:34] <ottomata>	 if i fail soon
[22:31:52] <ottomata>	 tomorrow i'll just make a new standalone puppetmaster node
[22:32:33] <Krenair>	 it still is able to point to a remote puppetmastyer
[22:32:35] <Krenair>	 puppetmaster
[22:34:30] <ottomata>	 yeah, i think i just busted it
[22:35:53] <Krenair>	 I'll deal with it
[22:37:12] <ottomata>	 Krenair:  don't worry about it, these nodes have been around for a while, and are only set up mostly for upgrade testing and puppet dev work
[22:37:42] <ottomata>	 all i need is a kafka broker on which i can test some puppet stuff.  i gotta quit for the day, so i'll just make a new one tomorrow, and delete a few of these
[22:37:44] <Krenair>	 I've been dealing with this kind of mess over the past week with puppetmasters
[22:37:46] <ottomata>	 will make a new node now so its ready for me tomorrow
[22:37:48] <ottomata>	 haha, ok
[22:37:50] <ottomata>	 if you can fix that's cool too
[22:37:53] <ottomata>	 i'll stop messing with it
[22:38:11] <ottomata>	 i think i just busted a bunch of cert stuff, i think i accidentally revoked/destroyed the puppetmaster cert on 401, and now stuff is bonkers :p
[22:39:19] <Krenair>	 okay it's working again
[22:39:23] <Krenair>	 well
[22:39:25] <Krenair>	 I mean
[22:39:27] <Krenair>	 it's past the SSL error
[22:39:33] <Krenair>	 you've still got Error: Could not retrieve catalog from remote server: Error 400 on SERVER: pick_initscript(): Wrong number of arguments given (6 for 7) at /etc/puppet/modules/base/manifests/service_unit.pp:90 on node kafka601.analytics.eqiad.wmflabs
[22:39:44] <ottomata>	 ah ok that changed i think i know why
[22:40:21] <ottomata>	 trying puppet on 601...
[22:40:46] <ottomata>	 yes!  puppet runnign, yeehaw
[22:40:48] <ottomata>	 thanks Krenair
[22:41:17] <Krenair>	 I basically got rid of the old /etc/puppet/puppet.conf.d/10-main.conf
[22:41:28] <Krenair>	 ran the recompilation of /etc/puppet/puppet.conf manually
[22:41:30] <ottomata>	 ah,i didin't see tha tone
[22:41:31] <ottomata>	 nice
[22:41:42] <Krenair>	 deleted the /var/lib/puppet/ssl and /var/lib/puppet/client/ssl directories
[22:41:47] <Krenair>	 cleaned the old cert from the master
[22:41:49] <Krenair>	 ran puppet
[22:42:09] <ottomata>	 agh, on second run it fails Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.
[22:42:13] <Krenair>	 Just not in that order, because I didn't remember all the things that needed doing and had to redo some
[22:42:46] <Krenair>	 what
[22:43:07] <ottomata>	 Notice: /Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/ensure: created
[22:43:12] <ottomata>	 in the first run
[22:43:34] <ottomata>	 Notice: /Stage[main]/Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/content: content changed '{md5}9f3978d4816ae16ad737cf46ca10af19' to '{md5}8914274be37e3e79178c73b212df781e'
[22:43:43] <ottomata>	 hehe
[22:43:43] <ottomata>	 Notice: /Stage[main]/Base::Puppet/Exec[delete master certs]: Triggered 'refresh' from 1 events
[22:43:44] <ottomata>	 Notice: /Stage[main]/Base::Puppet/Service[puppet]/ensure: ensure changed 'running' to 'stopped'
[22:43:56] <Krenair>	 yeah that'll screw things up
[22:43:58] <Krenair>	 why is it doing that
[22:44:08] <ottomata>	 role::puppet::self probably?
[22:44:50] <Krenair>	 role::puppet::self is supposed to *prevent* it from creating 10-main
[22:45:06] <ottomata>	 dunno!  it created it with 401 as master though
[22:45:11] <ottomata>	 dunno, ok Krenair thanks for you rhelp, i gotta run
[22:45:14] <Krenair>	 bye
[22:45:21] <ottomata>	 mucn appreciated, will try again tomorrow
[22:45:22] <ottomata>	 laters!
[22:49:01] <Zppix>	 how do you start a new kubectl job?
[22:58:38] <grrrit-wm>	 (03PS2) 10Paladox: Replacing swig with swig-templates [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320294 
[23:12:28] <Zppix>	 anyone know if javascript is support by jsub?
[23:30:30] <grrrit-wm>	 (03PS3) 10Paladox: Replacing swig with swig-templates [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320294 
[23:30:47] <grrrit-wm>	 (03PS4) 10Paladox: Replacing swig with swig-templates [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320294