[00:28:15] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1423561 (10Ricordisamoa) By the way, the 'old' static tool provides some logos that are not hosted on cdnjs: https://tools-static.wmflabs.org/static/logos/ [00:33:51] (03PS2) 10Yuvipanda: Allow addition of more root keys via hiera [labs/private] - 10https://gerrit.wikimedia.org/r/222468 [03:28:31] 6Labs, 10wikitech.wikimedia.org: Update Popups gadget in Wikitech - https://phabricator.wikimedia.org/T104676#1423772 (10MrStradivarius) 3NEW [05:41:11] 6Labs, 6Discovery, 10Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#1423929 (10Kolossos) No I change only the scripts we have in production inside Wikipedia and Commons. This doesn't also help maps that base on hikebikem... [06:40:15] 6Labs, 10Tool-Labs, 7HTTPS, 5HTTPS-by-default: Fix all http-only tools in tools.wmflabs.org - https://phabricator.wikimedia.org/T102457#1423989 (10Bugreporter) [06:55:59] (03CR) 10Merlijn van Deen: [C: 031] "one could consider" [labs/private] - 10https://gerrit.wikimedia.org/r/222468 (owner: 10Yuvipanda) [07:05:18] 10Tool-Labs-tools-Erwin's-tools: Fatal Error trying to use the related changes tool - https://phabricator.wikimedia.org/T104688#1424026 (10valhallasw) [08:50:01] hi guys! quick question. can I see the mysql error log somewhere? i'd like to understand better an error it is giving me. [08:53:25] marmick, mysql user errors are not logged [08:53:40] ok, thanks. [08:54:33] only server-wide errors like replication, starts, stops, etc. [09:49:46] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Shanmugamp7 was created, changed by Shanmugamp7 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Shanmugamp7 edit summary: Created page with "{{Tools Access Request |Justification=Stewards/crosswiki related tools and bots |Completed=false |User Name=Shanmugamp7 }}" [10:08:21] 10Quarry: SELECT command denied to user for table 'logging_userindex' - https://phabricator.wikimedia.org/T99414#1424205 (10PleaseStand) The query runs to completion if I change "USE commonswiki;" to "USE commonswiki_p;" (). So is this a problem with Quarry? [10:09:15] Hi there, I'm interested in helping cleanup spam and am new to Wikimedia Labs. [10:09:59] http://deployment.wikimedia.beta.wmflabs.org/wiki/User:Merseyside_user_5386 is my userpage [10:11:39] I would be interested in using temporary sysop status to test out things like abuse filter and any other gadgets etc. [10:24:44] 10Quarry: SELECT command denied to user for table 'logging_userindex' - https://phabricator.wikimedia.org/T99414#1424243 (10yuvipanda) 5Open>3Invalid a:3yuvipanda Nope, only *_p databases are available to users. commonswiki_p is the right thing to do. [10:26:08] 6Labs, 10Tool-Labs, 10Wikimania-Hackathon-2015: Conduct a research tools workshop at wikimania hackathon 2015 - https://phabricator.wikimedia.org/T91062#1424267 (10Qgil) Please confirm and promote this activity by assigning it to its owner, listing it or scheduling it at the [[ https://wikimania2015.wikimedi... [10:26:15] 6Labs, 10Tool-Labs, 10Wikimania-Hackathon-2015: Conduct a Tool Labs Workshop in Wikimania hackathon - https://phabricator.wikimedia.org/T91061#1424269 (10Qgil) Please confirm and promote this activity by assigning it to its owner, listing it or scheduling it at the [[ https://wikimania2015.wikimedia.org/wiki... [10:26:53] 10Tool-Labs-tools-Erwin's-tools: "Fatal Error: Database query failed" for Related Changes tool - MySQL down? - https://phabricator.wikimedia.org/T104688#1424294 (10Aklapper) [10:40:48] 6Labs: Provide a simple way to backup arbitrary files from instances - https://phabricator.wikimedia.org/T104206#1424327 (10yuvipanda) [10:42:00] 6Labs, 3Labs-Sprint-104: Recover files from old corrupted file system (Tracking) - https://phabricator.wikimedia.org/T104334#1424335 (10yuvipanda) [10:42:01] 6Labs, 10Tool-Labs-tools-meetbot, 10Labs-Infrastructure: Recover meetbot (toolslabs) files after NFS crash - https://phabricator.wikimedia.org/T104335#1424332 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Ok, so I found them in public_html, and have copied them all over to meetbot's $HOME/recovered. You c... [11:00:43] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103, 3Labs-Sprint-104: Labs: Salvage, then remove volumes on labstores' raid6 - https://phabricator.wikimedia.org/T103265#1424367 (10yuvipanda) https://phabricator.wikimedia.org/P871 is list of files that are irretrievably corrupted. [11:33:58] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL 60.00% of data above the critical threshold [0.0] [11:58:59] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK Less than 1.00% above the threshold [0.0] [14:10:58] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Shanmugamp7 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=168974 edit summary: [15:05:48] 10MediaWiki-extensions-OpenStackManager: Cant add a security group to an existing instance - https://phabricator.wikimedia.org/T42525#1424924 (10hashar) 5Open>3stalled That works in our OpenStack Horizon. So this bug is now just waiting for us to switch to Horizon. [15:05:56] 6Labs, 5Patch-For-Review: Make OpenStack Horizon useful for production labs - https://phabricator.wikimedia.org/T87279#1424927 (10hashar) [15:05:56] 10MediaWiki-extensions-OpenStackManager: Cant add a security group to an existing instance - https://phabricator.wikimedia.org/T42525#1424926 (10hashar) [15:40:25] 6Labs: Remove NFS mounts from project cephtest - https://phabricator.wikimedia.org/T102381#1424980 (10yuvipanda) I've deleted the project. Haven't removed it from the config list yet, since that'll be a good use to test how the script handles removals :) [17:04:18] 6Labs, 7Database: Provision a labsdb useraccount that can be used to run replica-addusers.pl - https://phabricator.wikimedia.org/T104476#1425172 (10jcrespo) I've just created a `labsdbadmin` user on mysql@labsdb100X with the minimum grants to execute for other users: ``` CREATE USER '${user_name}'@'%' IDENTIFI... [17:40:58] (03PS1) 10Sitic: Fix pt translation [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/222621 [17:41:23] (03CR) 10Sitic: [C: 032 V: 032] Fix pt translation [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/222621 (owner: 10Sitic) [17:54:34] 6Labs, 10Analytics-EventLogging, 6Performance-Team, 10VisualEditor, and 3 others: Pages can't be saved with VisualEditor on wikitech (Uncaught ReferenceError: attrId is not defined) - https://phabricator.wikimedia.org/T104360#1425384 (10Jdforrester-WMF) Yup, statsv complaint shows up but otherwise nothing... [18:34:55] YuviPanda: meh, too tired to work on mailrelay now. The DNS and hiera changes are safe to deploy, though. Not so for the MX change, but I have to think a bit about a better SPF setting [18:35:12] valhallasw`cloud: ok! [18:35:14] YuviPanda: imo sending and receiving mail servers should not necessarily be the same ones [18:35:25] I know literally nothing about mail :D [18:35:27] :D [18:35:56] Me neither, this is just from 'oh, by the way' comments from coren and scfc [18:36:01] right [20:18:18] (03CR) 10Incola: "Thanks for your time!" [labs/tools/maintgraph] - 10https://gerrit.wikimedia.org/r/222504 (owner: 10Ricordisamoa) [20:29:58] 6Labs: Login into fastcci-master instance not possible - https://phabricator.wikimedia.org/T104418#1425657 (10dschwen) 5Open>3Resolved a:3dschwen Since this morning access to fastcci-master seems to be restored. I'll close this ticket for now. [20:34:05] Hello! What's the command to restart the webserver if you are using fcgi? [20:40:29] valhallasw`cloud: I think I broke it again :( [20:41:23] error.log says: https://dpaste.de/12LF [20:44:00] webservice start [20:44:02] or something isn't it? [20:47:43] either webservice or webservice2 [20:47:54] but checking `status` on either tells me it ain't working [20:48:25] `webservice lighttpd status` says okey-dokey [20:48:45] But the tool is not actually running if you check the URL [20:52:18] ResMar: webservice lighttpd start [20:52:47] ResMar: error.log says your application has a g [20:52:49] Bug [20:52:52] `webservice lighttpd start` says it's running already---ah, so that *is* the command I should use [20:53:05] Restart, then. [20:53:21] But as I said, the issue is in your python code [20:54:01] It's possible I screwed something up, but app.py right now is just a hello world program [20:54:20] https://dpaste.de/wG5X [21:04:23] valhallasw`cloud: Does the spacing in error.log (https://dpaste.de/mim7) imply that the previous error is no longer present? ae. the fact that the server started several times after the most recent error message. [21:05:29] valhallasw`cloud: Or perhaps. from the FAQ: "Note that, as with the lighttpd setup, your tool will receive URL's that include your tool prefix - e.g. /MyToolName/index.html instead of /index.html. You may need to adapt your tool configuration to handle this." [21:07:54] Does that mean that to get to `https://tools.wmflabs.org/fc-importer/` I need `@app.route('fc-importer/')`, not `@app.route('/')` ? [21:08:54] ResMar: no, the fcgi config takes care of that [21:10:38] ResMar: huh, this is odd. [21:10:43] Fixed it. [21:10:55] Ugh. [21:10:57] oh, right. just a startup oddity [21:11:17] (qstat and webservice status didn't agree) [21:11:30] 30 minutes of running around restarting things because I can't write code for ****. [21:12:00] oh, you print hello world instead of returning it [21:12:01] :D [21:12:09] I also hadn't spotted that [21:12:20] Yeah, exactly [21:12:30] Looked at what I have running on my local virtual environment [21:12:33] Then looked really closely [21:12:37] Then, palm, forehead [21:13:04] Riiiight sorry about that [21:17:26] YuviPanda: oh, good, I can add multiple A records for the same domain. Let me test a new spf config then [21:18:43] valhallasw`cloud: Uh: so `webservice lighttpd restart` restores the link to working order, but for some reason only for a short time. ~30 seconds later the URL goes down again. [21:19:15] ResMar: did you just restart it? [21:19:22] It's down again now. [21:19:41] there's still a running lighttpd process according to qstat. let me kill thta [21:20:23] ResMar: ah! I think it's lighttpd and uwsgi fighting somehow [21:20:58] valhallasw`cloud: But isn't the uwsgi server off? [21:21:06] ResMar: it was still in the manifest somehow [21:22:35] valhallasw`cloud: I see the link is back up, should it be good to go now? [21:22:57] ResMar: yeah [21:24:14] Thanks! :) [21:29:12] 6Labs, 10Tool-Labs: Set up A-based SPF for tools.wmflabs.org - https://phabricator.wikimedia.org/T104733#1425752 (10valhallasw) 3NEW [21:29:21] SPF :-D [21:29:40] 6Labs, 10Tool-Labs: Set up A-based SPF for tools.wmflabs.org - https://phabricator.wikimedia.org/T104733#1425761 (10valhallasw) [21:29:46] SPF|Cloud: :D sorry [21:29:52] I know :p [21:29:55] It's okay [21:29:57] only so many 3 letter abbreviations possible [21:30:23] I can't think about a better abbreviation for South(P|p)ark(F|f)an [21:31:20] It might actually be time to learn what records like SPF do, since I only know how A and CNAME work :p [21:31:30] ZuiderParkLiefhebber? :D [21:31:43] that might give the wrong impression, though ;-) [21:32:28] SPF is a system to designate 'permitted senders', i.e. marking which servers may and may not send email with email addresses @ [21:32:51] so in the case of tools.wmflabs.org, only the tools mailserver is allowed to send mail, and all other mail should be marked as spam [21:35:06] 6Labs, 10Tool-Labs: Document process to run webserver on seperate instance through tools-webproxy - https://phabricator.wikimedia.org/T103056#1425766 (10valhallasw) [22:09:03] 6Labs, 10Tool-Labs: Document process to run webserver on seperate instance through tools-webproxy - https://phabricator.wikimedia.org/T103056#1425791 (10valhallasw) 5Open>3Resolved https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin/emergency_guides/single_tool_webservice [22:11:10] 6Labs, 10Tool-Labs: [tracking] Tool labs admin guides - https://phabricator.wikimedia.org/T104734#1425793 (10valhallasw) 3NEW [22:12:05] 6Labs, 10Wikibugs, 10grrrit-wm: document emergency deployment of grrrit-wm and wikibugs - https://phabricator.wikimedia.org/T102984#1425803 (10valhallasw) [22:12:06] 6Labs, 10Tool-Labs: Document process to run webserver on seperate instance through tools-webproxy - https://phabricator.wikimedia.org/T103056#1425802 (10valhallasw) [22:12:08] 6Labs, 10Tool-Labs: Audit redis usage on toollabs - https://phabricator.wikimedia.org/T91979#1425805 (10valhallasw) [22:12:10] 6Labs, 10Tool-Labs: Create process for 'tool labs is down' notifications on tools.wmflabs.org/* - https://phabricator.wikimedia.org/T102971#1425804 (10valhallasw) [22:12:12] 6Labs, 10Tool-Labs: [tracking] Tool labs admin guides - https://phabricator.wikimedia.org/T104734#1425801 (10valhallasw) [22:27:43] 6Labs, 10Tool-Labs: [tracking] Tool labs admin guides - https://phabricator.wikimedia.org/T104734#1425872 (10valhallasw) [22:27:44] 6Labs, 10Wikibugs, 10grrrit-wm: document emergency deployment of grrrit-wm and wikibugs - https://phabricator.wikimedia.org/T102984#1425870 (10valhallasw) 5Open>3Resolved https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin/emergency_guides/irc_bot_deployment [22:29:34] 6Labs, 10Analytics-EventLogging, 6Performance-Team, 10VisualEditor, and 3 others: Pages can't be saved with VisualEditor on wikitech (Uncaught ReferenceError: attrId is not defined) - https://phabricator.wikimedia.org/T104360#1425875 (10Krinkle) 5Open>3Resolved a:3Krinkle Can't reproduce this issues... [22:29:55] 6Labs, 10VisualEditor, 10VisualEditor-MediaWiki, 10wikitech.wikimedia.org, 7Regression: Pages can't be saved with VisualEditor on wikitech (Uncaught ReferenceError: attrId is not defined) - https://phabricator.wikimedia.org/T104360#1425878 (10Krinkle) [22:31:58] 6Labs, 10VisualEditor, 10VisualEditor-MediaWiki, 10wikitech.wikimedia.org, 7Regression: Pages can't be saved with VisualEditor on wikitech (Uncaught ReferenceError: attrId is not defined) - https://phabricator.wikimedia.org/T104360#1425880 (10Jdforrester-WMF) 5Resolved>3Open Still trivially reproduci... [22:36:37] 6Labs, 10VisualEditor, 10VisualEditor-MediaWiki, 10wikitech.wikimedia.org, 7Regression: Pages can't be saved with VisualEditor on wikitech (Uncaught ReferenceError: attrId is not defined) - https://phabricator.wikimedia.org/T104360#1425886 (10Krenair) Why didn't https://gerrit.wikimedia.org/r/#/c/221969/... [22:37:19] 6Labs, 10VisualEditor, 10VisualEditor-MediaWiki, 10wikitech.wikimedia.org, 7Regression: Pages can't be saved with VisualEditor on wikitech (Uncaught ReferenceError: attrId is not defined) - https://phabricator.wikimedia.org/T104360#1425888 (10Jdforrester-WMF) p:5Triage>3Low