[00:02:15] 06Labs, 10Phabricator, 07Puppet: Phabricator labs puppet role configures phabricator wrong - https://phabricator.wikimedia.org/T131899#2995148 (10Paladox) 05Open>03Resolved a:03Paladox yep, this works now. [00:02:27] 06Labs, 10Phabricator, 07Puppet: Phabricator labs puppet role configures phabricator wrong - https://phabricator.wikimedia.org/T131899#2995151 (10Paladox) a:05Paladox>03None [00:05:38] 06Labs, 10Phabricator, 13Patch-For-Review: Applying role role::phabricator::main causes errors on instances - https://phabricator.wikimedia.org/T138881#2412999 (10Paladox) This all works in labs now. Just need to do https://gerrit.wikimedia.org/r/335703 [00:06:19] 06Labs, 10Labs-project-Phabricator, 13Patch-For-Review: Applying role role::phabricator::main causes errors on instances - https://phabricator.wikimedia.org/T138881#2995171 (10Paladox) [00:06:30] 06Labs, 10Labs-project-Phabricator, 10Phabricator, 13Patch-For-Review: Applying role role::phabricator::main causes errors on instances - https://phabricator.wikimedia.org/T138881#2412999 (10Paladox) [04:09:23] Anyone around? [06:07:36] 06Labs, 10Tool-Labs: templatetiger is using 613G in Tools out of 8T - https://phabricator.wikimedia.org/T136192#2995607 (10Bgwhite) I removed any of 2016's CheckWiki output files. I kept 2017's files. It should be much lower now. [06:36:59] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:44:21] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:47:28] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:59:37] PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:11:59] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:28] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [07:24:21] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [07:33:37] 10Labs-project-Wikistats, 07I18n: Gõychi Konknni's English Wikistats translation is incorrect - https://phabricator.wikimedia.org/T156814#2995688 (10Nemo_bis) [07:36:09] 10Labs-project-Wikistats, 07I18n: Gõychi Konknni's English Wikistats translation is incorrect - https://phabricator.wikimedia.org/T156814#2986727 (10Nemo_bis) The actual WikiStats used to fetch names from Names.php, IIRC. mutante's wikistats (which is what you're reporting about) might fetch them from the API. [07:39:38] RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [07:53:47] 10Labs-project-Wikistats, 07I18n: mutante's Wikistats uses the old English name of Gõychi Konknni - https://phabricator.wikimedia.org/T156814#2995716 (10Nemo_bis) [08:47:59] new tool: https://tools.wmflabs.org/ldap/ [08:53:45] legoktm: I find some no-longer-wmfers in https://tools.wmflabs.org/ldap/group/wmf [08:54:55] zhuyifei1999_: hm, that should probably be filed as a bug for #operations to take care of, my tool is just pulling from the live ldap server [08:55:14] yeah ik [08:56:01] ldap list probably needs faster updates, similar to admin.yaml [09:38:33] 06Labs, 10Tool-Labs: "webservice restart" fails when using kubernetes - https://phabricator.wikimedia.org/T157109#2995863 (10Legoktm) [09:42:53] 06Labs, 10Tool-Labs: "webservice restart" fails when using kubernetes - https://phabricator.wikimedia.org/T157109#2995882 (10Legoktm) Workaround for now: ``` tools.wikitext-deprecation@tools-bastion-03:~/www/python/src$ webservice stop Stopping webservice tools.wikitext-deprecation@tools-bastion-03:~/www/pytho... [09:50:11] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#2995891 (10zhuyifei1999) [09:50:13] 06Labs, 10Tool-Labs: "webservice restart" fails when using kubernetes - https://phabricator.wikimedia.org/T157109#2995893 (10zhuyifei1999) [09:55:05] PROBLEM - High iowait on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: tools.tools-webgrid-lighttpd-1413.cpu.total.iowait (>11.11%) [09:59:23] zhuyifei1999_: whoops, thanks [09:59:35] np [10:05:06] RECOVERY - High iowait on tools-webgrid-lighttpd-1413 is OK: OK: All targets OK [10:21:48] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#2996017 (10Marostegui) db1064 the future master of db1095 for s4 has now been: - Changed its binary log to ROW base - Upgraded from 10.0.23 to 10.0.29 (an... [11:30:17] 06Labs, 10Labs-Infrastructure, 10DBA: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869#2996136 (10Marostegui) Hi, By no means what I am going to say is the way to go, but just expressing my opinion and showing a method we are doing in produc... [12:05:28] 06Labs, 10DBA, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#2996223 (10Marostegui) db1095 has finished transferring its snapshot to: es1017:/srv/tmp I have tested to decompress it and it worked fine. db1095 is bac... [13:44:18] Anyone around to modify my approved OAuth consumer registration? [14:05:02] JustBerry: I believe bd808 can but and he'll be around in a bit [14:05:13] thanks! chasemp [15:07:04] 10Tool-Labs-tools-Xtools, 06Community-Tech: Migrate XTools from Ubuntu Precise to Trusty - https://phabricator.wikimedia.org/T157123#2996519 (10JJMC89) [15:16:37] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, and 2 others: Remove support for precise OGE exec hosts - https://phabricator.wikimedia.org/T94792#2996566 (10zhuyifei1999) [15:16:40] 10Tool-Labs-tools-Xtools, 06Community-Tech: Migrate XTools from Ubuntu Precise to Trusty - https://phabricator.wikimedia.org/T157123#2996565 (10zhuyifei1999) [15:56:23] o/ JustBerry [15:56:39] halfak: Okay, let me post my question in here. I'll have to compile all the PMs lol [15:56:48] I heard you have some flask on tool labs issues. I'm not sure I can help, but I'm sure someone here can :) [16:00:25] I'm making a script that takes in an article name and prints a series of strings on different lines. I realize that I need to change the print statements to return statements to the flask framework, of course. I've got the consumer registration done, and the app.py seems to be able to successfully connect to the OAuth registration (finished setup instructions here https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Python_appl [16:00:25] ication_stub). Also, the fool is up here: https://tools.wmflabs.org/spiarticleanalyzer. Reading through the flask docs, I know I have to do @app.route("/"). I currently have my code on GitHub (a directory with multiple files). I have a main script that returns what needs to be returned and fellow helper .py files which are referenced in the main .py file. My question is a) which directory do I put my code in tool labs (which [16:00:25] I can successfully ssh into and "become") and b) can I put the @app.route("/") methods anywhere in that main .py file (substituting the print statements for return statements as presented in the flask docs here: http://flask.pocoo.org/)? [16:00:29] halfak: ^^ ;p [16:00:52] fool => tool* [16:01:56] link to github repo? [16:03:17] https://github.com/JustBerry/SPIArticleAnalyzer @ halfak [16:06:29] I don't see any flask imports [16:07:15] halfak: haven't [16:07:25] @ halfak b) can I put the @app.route("/") methods anywhere in that main .py file (substituting the print statements for return statements as presented in the flask docs here: http://flask.pocoo.org/)? [16:07:28] ^^ [16:07:46] JustBerry, which is the "main py file" [16:07:52] I don't see you constructing an app [16:07:55] I might be missing it [16:08:08] halfak: ah, whops. it's getAllUsers.py [16:08:14] I have to convert it into an app [16:08:46] from flask import Flask [16:08:47] app = Flask(__name__) [16:08:52] You need this in that file [16:09:04] then you *use* app.route() later in the file [16:09:39] Yuck, python2 [16:10:03] yeryry, agreed. [16:23:19] python2 4evr ;) [16:25:09] JustBerry: The instructions at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#Python_2_.28uWSGI.29 give you the basics that are needed. Code goes in $HOME/www/python/src with a $HOME/www/python/src/app.py module that is the flask wsgi app itself. [16:25:27] The associated virtualenv belongs in $HOME/www/python/venv [16:26:10] halfak: I updated it in the code [16:26:11] This tool is a pretty basic example -- https://phabricator.wikimedia.org/source/tool-precise-tools/ [16:26:15] bd808: okay, thanks... let me take a look [16:26:19] halfak: (just pushed a new commit) [16:27:00] "def display(output):" won't work. [16:27:07] the "output" param doesn't come anywhere [16:27:09] *from [16:27:35] change it to "display(): return "hello" [16:28:28] halfak: how can I return the string I put together then [16:28:33] the output string [16:28:48] method and string both called output [16:28:54] You'll need to generate it within that function call. [16:28:58] o/ [16:29:20] @app.route("/") [16:29:34] def display(article_name): ... [16:30:56] halfak: better now? [16:31:05] the entire code... hm... [16:31:13] I see. [16:31:15] hold on. [16:31:19] (will change) [16:31:57] 06Labs, 10Tool-Labs, 10DBA: u3532__ (=marcmiquel) table using 64G on labsdb1001 - https://phabricator.wikimedia.org/T133322#2228158 (10Huji) @marcmiquel are you done with the clean up? If yes, would you mind closing this task? [16:35:43] halfak: literally "@app.route("/")"? p.s. pushed changes per your prior recommendation [16:36:59] Why have you called the function with no arguments at the end? [16:38:02] halfak: done [16:38:18] Looks like a flask app to me [16:38:24] halfak: hhe [16:38:26] heh* [16:38:31] lemme try to get it up now I suppose [16:38:39] I haven't tested it, so mind the tiny errors like the one you mentioend [16:39:03] halfak: btw, git clone creates a whole folder [16:39:20] I suppose the only way to get only the raw files downloaded to the src dir, I'll need to manually do each one [16:40:03] No idea how tools works with a flask app [16:40:11] halfak: I'll try git fetch [16:48:32] bd808: for webservice --backend=kubernetes python2 start, bash: webservice: command not found [16:48:36] running from ~ [16:49:17] ugh. yuvipanda was working on some error like that the other day [16:49:42] the other one was working (let me think what I used) [16:49:46] webservice command not found? [16:49:55] which host are you on? [16:50:04] this works bd808 webservice2 uwsgi-python start [16:50:11] zhuyifei1999_: in the ~ dir [16:50:16] host? [16:50:22] host, not directory [16:50:26] idk [16:50:38] how do I check? [16:50:48] JustBerry: what does `hostname -f` say? [16:50:58] bd808: interactive [16:51:04] what's your prompt? the thing before you type your command [16:51:05] zhuyifei1999_: ^^ [16:51:12] tools.spiarticleanalyzer@interactive:~$ [16:51:15] that's on k8s [16:51:27] o [16:51:30] I'm back on bastion [16:51:31] whops lol [16:51:34] my bad [16:51:37] you need to exit that before starting webservice [16:51:51] that command is only on bastions [16:52:19] zhuyifei1999_: yeah [16:52:31] (i.e. tools-bastion-0) [16:53:05] zhuyifei1999_: you have the channel history, right? [16:53:16] yeah [16:53:17] I have to link the .py script (converted to app) [16:53:24] I put the app method [16:53:29] @app. etc [16:53:36] https://github.com/JustBerry/SPIArticleAnalyzer/blob/master/getAllUsers.py [16:54:10] * zhuyifei1999_ looks (will take a while, I'm on mobile) [16:54:36] bd808: do I just run the py script... [16:54:42] but it has to be user-based [16:54:44] whops [16:54:48] end-user activated I meant [16:54:52] i.e. end-user inputs something [16:54:58] end-user can't access tool, so meh [16:57:12] eh, need pip installed by a labs admin [16:57:43] JustBerry: let me find the virtualenv instructions for you... [16:58:06] thanks again [16:59:05] JustBerry: for user input, you need a form [16:59:18] zhuyifei1999_: yeah [16:59:27] well firstly, I can't test out the script [16:59:32] even w/o user input, as I need pip [16:59:36] pip install a module [16:59:45] prolly need an admin? [17:00:01] use pip in virtualenv [17:00:07] https://wikitech.wikimedia.org/wiki/Setting_up_Flask_cgi_app_as_a_tool#Set_up_virtualenv [17:00:25] bd808: ^^ [17:00:34] and the form directs the user to another url [17:00:56] that includes a GET or POST param [17:01:10] @tools-bastion-03:~$ pip install geoip2 [17:01:16] not supposed to run on bastion? [17:01:43] and handle that url with another app.route [17:01:46] yes, but you enter the virtualenv first [17:02:06] bd808: i did @tools-bastion-03:~$ virtualenv python [17:02:13] um, it's preferred to mess with virtualenv within k8s [17:02:21] just does [17:02:23] @tools-bastion-03:~$ virtualenv python [17:02:30] New python executable in python/bin/python [17:02:30] Installing setuptools, pip...done. [17:02:31] that [17:02:34] and then back to bastion again [17:02:35] different environments, you know [17:02:45] JustBerry: source ~/www/python/venv/bin/activate [17:03:09] um idk why are you doing $ virtualenv python [17:03:20] virtualenv is a python thing [17:03:31] ok [17:03:52] zhuyifei1999_: but wait, the module is for my script [17:03:53] other languages have different methods to isolate environments [17:03:58] not for flask [17:04:01] yeah [17:04:06] still didn't show up [17:04:06] ImportError: No module named geoip2.database [17:04:07] ? [17:04:09] even after all that [17:04:35] ah okay [17:04:36] wait [17:04:45] have you done the activate thing bd808 said? [17:04:49] zhuyifei1999_: yes [17:04:59] okay, seems to be working, resolving code bugs now [17:05:09] pip installed the module? [17:07:17] zhuyifei1999_: yes [17:07:24] while debugging, this: [17:07:27] "An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform." [17:07:40] InsecurePlatformWarning [17:07:47] ignore that [17:08:00] that's during pip install right? [17:08:05] zhuyifei1999_: not sure if it did anything [17:08:06] no after [17:08:09] that's when I ran my script [17:08:14] and it went inside the display() function [17:08:19] something happened there [17:08:21] with the displaying [17:08:32] I mean the python script works fine, so had to be the display part [17:08:52] zhuyifei1999_: should I paste the entire error [17:08:56] in pastebin [17:09:02] hmm I never understand what the error is about [17:09:23] http://pastebin.com/pHQUQ5Gx [17:09:43] maybe bd808 or halfak knows heh [17:10:00] No idea what that is. [17:10:20] JustBerry: you can ignore the InsecurePlatformWarning [17:10:37] 10Labs-project-Wikistats, 07I18n: mutante's Wikistats uses the old English name of Gõychi Konknni - https://phabricator.wikimedia.org/T156814#2996946 (10Niedzielski) Thank you @Nemo_bis, @Nikerabbit! (Sorry for the confusion @Milimetric) It looks like [[ https://phabricator.wikimedia.org/rMWb49c4503070dbdf6b8... [17:10:41] um idk why I'm having trouble accessing pastebin, can you paste it in phab? [17:11:16] hi, I have a quick quick question about accessing the replica sql database -- the documentation says the username+passwd should be in the replica.my.cnf file, but I can't find it [17:11:35] it's probably only printing to stderr, so won't affect the output of the webpage [17:11:49] stefano___: your account doesn't have a replica.my.cnf at all? [17:12:04] zhuyifei1999_: https://phabricator.wikimedia.org/P4881 [17:12:07] can't find it in my $HOME folder [17:12:27] stefano___: I believe they re only generated for tool homes not individual users atm [17:12:30] bd808: doesn't seem to be working on the tool... I'm assuming the string I returned SHOULD be showing up at https://tools.wmflabs.org/spiarticleanalyzer/, unless I'm mistaken [17:12:46] zhuyifei1999_: let me know if there are any issues accessing the paste [17:13:38] yeah just ignore that, it'll only appear in your logs for webservices [17:13:47] uwsgi.log [17:14:44] JustBerry: btw, remember to restart webservices after each code update [17:14:58] ay [17:16:43] zhuyifei1999_: er... from venv? [17:17:15] no, just $ webservice restart [17:17:21] from bastion [17:17:25] zhuyifei1999_: aight [17:18:05] zhuyifei1999_: nothing displayed tbh [17:18:28] I'll note that operating venv within k8s and within bastion may produce different results [17:19:01] zhuyifei1999_: nothing o/ https://tools.wmflabs.org/spiarticleanalyzer/ [17:19:41] anything in your uwsgi.log? [17:20:42] 06Labs, 10Tool-Labs: zoomviewer is using 837G out of 8T for Tools - https://phabricator.wikimedia.org/T136190#2996989 (10madhuvishy) Thank you @dschwen! [17:20:49] zhuyifei1999_: I'll check, but it DOES run completely [17:20:54] I have a point in my code where I check that [17:20:55] and where's your app.py? [17:20:57] uh [17:21:05] app.py is the parent directory [17:21:16] because I had to git clone [17:21:26] what? o.O [17:21:27] so the repo's in the same dir as the app.py [17:21:31] okay wait [17:21:35] src: app.py, REPO [17:21:42] src/REPO: thescript.py [17:21:56] now let me check what you said about uwsgi.log [17:22:24] btw, is the log in the same folder as app.py? it's empty, if so [17:22:37] I did nano uwsgi.log in the app.py dir [17:22:53] app.py should be a python file [17:23:03] zhuyifei1999_: yes [17:23:11] the uwsgi.log file will be in the tool's home dir [17:23:13] the file that has the flask app in it [17:23:13] but not in the same dir as the script I'm running [17:23:19] not a dir... [17:23:35] .... [17:23:40] it's a file [17:23:41] not a dir [17:23:42] I know [17:23:43] app.py [17:24:04] bd808: do you follow what I mean about the location of the app.py versus the script .py? [17:24:13] so what's in it? I can't see the file in your repo? [17:24:14] www/python/src: app.py [17:24:20] app.py should be in my repo? [17:24:29] and clone the repo into src? [17:25:41] um maybe I should try fire up a ssh shell (in mobile) to see what's going on... [17:25:54] zhuyifei1999_: in mobile o/ [17:29:47] JustBerry: look at the README for https://phabricator.wikimedia.org/source/tool-precise-tools/ and see if it helps you make sense of things. [17:29:55] bd808: ok [17:30:03] bd808: that's what I used [17:30:18] JustBerry: your app.py is a chmoded-600 regular file [17:30:19] but let me look at the file setup [17:30:22] teh redis-prefix stuff there is unrelated (its app specific config) [17:30:27] o [17:30:31] I did the redis stuff [17:30:31] .. [17:30:42] it won't hurt anything [17:31:15] bd808: you excluded the keys from the file? [17:31:25] probably shouldn't push the keys to github ;p [17:31:38] which keys? [17:31:44] bd808: consumer_key [17:32:00] should be in app.py... [17:32:06] not here, not sure why https://phabricator.wikimedia.org/source/tool-precise-tools/browse/master/app.py [17:32:22] Your OAuth secrets should *not* be in version control [17:32:35] the precise-tools app has no secrets really [17:32:59] bd808: how would you recommend putting those aside then [17:33:15] the instructions said to have the oauth details right in app.py [17:33:20] in some sort of configuration file [17:33:22] you should separate your config secrets into a file and read them at startup [17:33:33] and have the app read from the file [17:33:52] "the instructions" are very wrong then... is that that one of the user created tutorials? [17:34:23] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Python_application_stub step 3 @ bd808 [17:35:45] I'll see if I can fix that up a bit later today [17:37:49] * zhuyifei1999_ meh [17:38:28] I'm probably not gonna play with a chmodded 600 stuff [17:38:48] I mean code, not config file [17:44:40] zhuyifei1999_: I did all that [17:44:44] works [17:44:51] i.e. the config stuff [17:44:52] so [17:44:53] https://github.com/JustBerry/SPIArticleAnalyzer/blob/master/app.py [17:44:58] it's all there now [17:45:00] in the repo [17:47:26] zhuyifei1999_: bd808 so everything's under @tools-bastion-03:~/www/python/src/SPIArticleAnalyzer now... including the app.py, requirements.txt,
.py, whatnot [17:47:48] when I run
.py (which is getAllUsers.py)... doesn't show up at https://tools.wmflabs.org/spiarticleanalyzer/ still (login OR logout) [17:47:50] app.py needs to be at ~/www/python/src/app.py [17:47:58] bd808: so do the other files then? [17:48:01] where should the other files be [17:48:10] ok so a few issues: [17:48:17] amn't I supposed to have app.py in the repo [17:48:21] then where does the repo go [17:48:26] @ bd808 [17:48:27] zhuyifei1999_: yes? [17:48:34] 1. why are all those random comments and prints? [17:48:59] zhuyifei1999_: which [17:49:15] in the app.py? [17:49:25] 2. I'm pretty sure file.readline() returns a string with a line ending at the end, you might want to handle that [17:49:29] that's what I downloaded from... wget https://raw.githubusercontent.com/valhallasw/flask-mwoauth/master/demo.py -O app.py [17:49:38] 3. file is not closed properly [17:50:11] JustBerry: there seems to be quite a bit of confusion here. The python wsgi setup we have is quite opinionated. I think I need to update some documentation to be more readable and understandable rather than trying to walk you though step by step. [17:50:12] 4. you have an index, which does not call the rest of the code at all... [17:51:49] there are many flask setups you can refer to [17:52:39] bd808: sounds good [17:52:49] zhuyifei1999_: let me address those four points [17:52:50] thanks [17:53:04] https://github.com/toollabs/video2commons/ repo dir is home dir [17:53:44] zhuyifei1999_: is that www/python/src/video2commons? [17:54:04] other way around [17:54:08] video2commons/www/python/src [17:54:16] ? [17:56:02] the app.py https://github.com/toollabs/video2commons/blob/master/www/python/src/app.py is simply import app from elsewhere [17:57:35] btw, for closing file, I suggest using with statement [17:58:23] context managers are awesome [17:58:25] 06Labs, 10Striker, 10Tool-Labs, 05Security: striker does not (?) honour TitleBlacklist for shell names - https://phabricator.wikimedia.org/T156343#2997075 (10bd808) [18:14:24] PROBLEM - Free space - all mounts on tools-worker-1004 is CRITICAL: CRITICAL: tools.tools-worker-1004.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found)tools.tools-worker-1004.diskspace.root.byte_percentfree (<60.00%) [18:21:07] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/S.desabbata was modified, changed by S.desabbata link https://wikitech.wikimedia.org/w/index.php?diff=1453024 edit summary: [18:21:41] zhuyifei1999_: is this better https://github.com/JustBerry/SPIArticleAnalyzer [18:22:18] looking [18:23:33] oh my god [18:23:44] definitely not [18:24:04] don't commit *everything* [18:24:05] (03PS1) 10Mholloway: Add time zone to alpha build timestamp on android-builds.wmflabs.org [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/335845 (https://phabricator.wikimedia.org/T108318) [18:24:18] zhuyifei1999_: .. [18:24:21] isn't that what you did o/ [18:25:01] bd808: ^ is beyond my ability to fix [18:25:30] JustBerry: I don't commit *everything* [18:25:34] oops. ok [18:26:10] ;/ [18:26:22] bd808: was that not.... [18:26:40] JustBerry: you have made several secrets public there. [18:26:44] we can fix is [18:26:46] *it [18:26:54] well [18:27:40] once a secret is out there on github there is no taking it back, but we can rotate them to new values [18:28:33] bd808: deleted the repo [18:28:58] yeah. we will still need to get you new database and kubernetes secrets [18:29:10] oy [18:29:20] not the end of the world [18:29:32] JustBerry: anyways I didn't /tell/ you to commit the home dir, what I said was that's one way to do it [18:29:58] oh and no point in committing the virtualenvs [18:31:25] and if the oauth text was in that repo, you gotta create a new one. compromised creds gotta be disabled [18:31:47] zhuyifei1999_: no no [18:31:49] I had that in .gitignore [18:31:56] I thought the other stuff was fine eh [18:32:29] It's all ok JustBerry. Mistakes like this happen. [18:32:35] (meh) [18:32:57] bd808: do I have to use the commit scrubber now [18:33:02] and recommit the last "working" version ;p [18:33:21] I guess so [18:33:30] https://help.github.com/articles/removing-sensitive-data-from-a-repository/ [18:33:31] heh [18:33:38] let's just stop for a bit and get the secrets fixed before going father [18:34:09] bd808: aight [18:34:11] * bd808 is looking for instructions on how to handle the secret changes [18:34:16] bd808: want to PM maybe o/ [18:34:32] you can't do it. this is a root thing :) [18:38:35] !log tools.spiarticleanalyzer Removed ~/.kube as root [18:38:36] Unknown project "tools.spiarticleanalyzer" [18:38:51] oh come on stashbot [18:40:08] stashbot: screw you ;p [18:40:09] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [18:41:38] tgr: o/ [18:42:54] hi JustBerry [18:43:19] is the reason you want the prefix flag enabled compatibility with some sort of framework? [18:43:33] I'd much rather fix that issue in our OAuth server [18:43:52] ie https://phabricator.wikimedia.org/T156187 [18:44:05] tgr: wait what [18:44:51] 06Labs, 10Tool-Labs: Rotate mysql and k8s secrets for tools.spiarticleanalyzer - https://phabricator.wikimedia.org/T157158#2997194 (10bd808) [18:45:01] 06Labs, 10Tool-Labs: Rotate mysql and k8s secrets for tools.spiarticleanalyzer - https://phabricator.wikimedia.org/T157158#2997207 (10bd808) p:05Triage>03Unbreak! [18:45:08] Is there any documentation on labs backups somewhere? [18:45:23] Samwalton9: wew [18:45:35] ? [18:45:46] Samwalton9: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Backups [18:45:47] ? [18:46:02] Ah, thanks [18:46:17] Samwalton9: basically, there are no Labs backups. At least none that you can rely on as a Labs user. [18:46:48] This is something that we would like to fix, but its a ways down on the priority list [18:47:07] Ok great, just wondering if we need to prioritise that, which it sounds like we do :D [18:47:43] 'all' we need is more hardware and more people to operate it ;) [18:47:44] JustBerry: I mean the only difference between https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/44b3c78e8190647f672eea4c612f471a and https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/fa88fc9efa0e08429863160d038f70a9 is that flag [18:47:51] and it seems like a pointless change [18:48:01] tgr: needed the private key too [18:49:15] you can always change private keys via https://meta.wikimedia.org/wiki/Special:OAuthConsumerRegistration/list > manage > update [18:50:34] JustBerry: FYI, you can test out a consumer you proposed while it hasn't been approved [18:51:53] zhuyifei1999_: yeah [18:52:03] tgr: ah [19:05:22] bd808: about that stashbot error, remember that it didn't know about tools.precise-tools? maybe it could be the same error? [19:06:06] zhuyifei1999_: yeah. I found the bug this time too :) [19:06:35] my cache invalidation test used exp_time > now rather than exp_time < now ! [19:06:54] lol [19:11:42] 06Labs, 10Tool-Labs: Rotate mysql and k8s secrets for tools.spiarticleanalyzer - https://phabricator.wikimedia.org/T157158#2997194 (10chasemp) ``` /usr/local/sbin/maintain-dbusers delete tools.spiarticleanalyzer --debug Deleted account in labsdb1009.eqiad.wmnet for tools.spiarticleanalyzer Deleted account in l... [19:11:57] 06Labs, 10Tool-Labs: Rotate mysql and k8s secrets for tools.spiarticleanalyzer - https://phabricator.wikimedia.org/T157158#2997283 (10chasemp) a:03yuvipanda [19:12:18] 06Labs, 10Tool-Labs: Rotate mysql and k8s secrets for tools.spiarticleanalyzer - https://phabricator.wikimedia.org/T157158#2997194 (10chasemp) tossing your way @yuvipanda to rotate the k8s secret [19:26:23] (03PS1) 10BryanDavis: Tweak git config [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/335851 [19:26:26] (03PS1) 10BryanDavis: Fix LDAP cache expiration check [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/335852 [19:26:54] (03CR) 10BryanDavis: [C: 032] Tweak git config [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/335851 (owner: 10BryanDavis) [19:26:56] (03CR) 10jerkins-bot: [V: 04-1] Fix LDAP cache expiration check [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/335852 (owner: 10BryanDavis) [19:27:58] (03Merged) 10jenkins-bot: Tweak git config [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/335851 (owner: 10BryanDavis) [19:28:23] !log mobile created readinglists instance [19:28:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL [19:30:05] (03PS2) 10BryanDavis: Fix LDAP cache expiration check [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/335852 [19:31:08] (03CR) 10BryanDavis: [C: 032] Fix LDAP cache expiration check [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/335852 (owner: 10BryanDavis) [19:31:46] (03Merged) 10jenkins-bot: Fix LDAP cache expiration check [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/335852 (owner: 10BryanDavis) [19:35:41] !log tools.stashbot Updated to 6f45ed3 Fix LDAP cache expiration check (T156652) [19:35:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [19:35:44] T156652: Stashbot not seeing tools.precise-tools as a valid project - https://phabricator.wikimedia.org/T156652 [19:36:15] 10Tool-Labs-tools-Other, 15User-bd808: Stashbot not seeing tools.precise-tools as a valid project - https://phabricator.wikimedia.org/T156652#2997319 (10bd808) 05Open>03Resolved Fixed with https://gerrit.wikimedia.org/r/#/c/335852/ [19:41:39] 06Labs, 10Tool-Labs: Rotate mysql and k8s secrets for tools.spiarticleanalyzer - https://phabricator.wikimedia.org/T157158#2997325 (10yuvipanda) 05Open>03Resolved Done. To regen k8s creds: 1. Remove acct line from `/etc/kubernetes/tokenauth` 2. Run `systemctl restart maintain-kubeusers` [19:44:47] 10Tool-Labs-tools-Xtools, 06Community-Tech: Migrate XTools from Ubuntu Precise to Trusty - https://phabricator.wikimedia.org/T157123#2996293 (10MusikAnimal) I think the jobs that are still running on Trusty are for [[ https://tools.wmflabs.org/xtools/wikihistory/ | WikiHistory ]]. This is a tool chiefly used b... [19:45:08] bd808: so o/ [19:45:55] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#2997335 (10kaldari) [19:45:59] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Investigation: Plan for rewriting XTools - https://phabricator.wikimedia.org/T154551#2997333 (10kaldari) 05Open>03Resolved Thanks for the thorough investigation! [19:46:13] JustBerry: I'm working on updating https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Python_application_stub -- I think that making things more clear there is the best next step. [19:46:22] bd808: ok [19:46:40] It's hard to do this over irc without a good document to reference [19:47:05] but on the plus side, this is something I have 'been meaning to do' for a long time :) [19:47:31] * bd808 -> foodz [19:49:01] 06Labs, 10Tool-Labs: Rotate mysql and k8s secrets for tools.spiarticleanalyzer - https://phabricator.wikimedia.org/T157158#2997367 (10yuvipanda) Documented at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin#Regenerate_kubernetes_credentials_.28.kube.2Fconfig.29 [20:12:58] bd808: rotated. [20:26:04] zhuyifei1999_: I should probably wait to test even though the keys are rotated according to the ticket? [20:26:14] plus tutorial is being written so [20:48:08] 10Labs-project-Wikistats, 07I18n: mutante's Wikistats uses the old English name of Gõychi Konknni - https://phabricator.wikimedia.org/T156814#2997514 (10Dzahn) a:03Dzahn I'll look into it but it's unrelated to Names.php. Those names are just in a local database on a labs instance. [20:49:22] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:55:02] !log mobile apply role::labs::mediawiki_vagrant to instance readinglists [20:55:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL [20:57:12] !log mobile apply role::restbase::server to instance readinglists [20:57:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL [21:06:32] !log mobile remove role::restbase::server from instance readinglists [21:06:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL [21:13:52] !log tools reboot tools-bastion-03 as unresponsive [21:13:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:14:50] who's rush chasemp [21:14:54] me [21:29:22] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:34:31] !log tools Migrated over precise tools to trusty for user multichill (catbot, family, locator, multichill, nlwikibots, railways, wlmtrafo, wikidata-janitor) [21:34:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:05:08] (03PS1) 10Jean-Frédéric: Add unit test for populate_image_table.getSources [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/335935 [22:20:33] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Dump instance info as a static file updated periodically - https://phabricator.wikimedia.org/T143136#2557989 (10Andrew) Is this moot now that we have api observer access? [22:21:19] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Don't rely on wikitech API for production services - https://phabricator.wikimedia.org/T104575#2997719 (10Andrew) 05Open>03Resolved a:03Andrew I think that's all of them! [22:36:32] 10Tool-Labs-tools-Pageviews, 06Community-Tech, 10GLAM-Tech: Massviews: bug with pagepile adding extra Wikipedia: on GLAM cultural partnership project pages - https://phabricator.wikimedia.org/T135437#2298812 (10Tbayer) T156993 may be interesting in this context. [22:52:10] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Dump instance info as a static file updated periodically - https://phabricator.wikimedia.org/T143136#2997778 (10bd808) >>! In T143136#2997707, @Andrew wrote: > Is this moot now that we have api observer access? Maybe it needs to be repurposed to document us... [23:13:16] 10Labs-project-Wikistats, 07I18n: mutante's Wikistats uses the old English name of Gõychi Konknni - https://phabricator.wikimedia.org/T156814#2997803 (10Dzahn) ``` MariaDB [(none)]> use wikistats; Reading table information for completion of table and column names You can turn off this feature to get a quicker... [23:14:20] 10Labs-project-Wikistats, 07I18n: mutante's Wikistats uses the old English name of Gõychi Konknni - https://phabricator.wikimedia.org/T156814#2997804 (10Dzahn) 05Open>03Resolved @Niedzielski Fixed. Should be as you requested now. https://wikistats.wmflabs.org/detail.php?t=wp&id=307 [23:15:51] 10Labs-project-Wikistats, 07I18n: mutante's Wikistats uses the old English name of Gõychi Konknni - https://phabricator.wikimedia.org/T156814#2997808 (10Niedzielski) @Dzahn, thank you!! [23:35:32] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 2 others: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#1344508 (10srishakatux) This project is selected for the #developer-wishlist voting round and will be added to a MediaWiki page very soon. To...