[00:00:18] there is no such port forwarding because that's not where it would connect normally [00:00:51] by default it uses the xdebug multi-developer setup in which it connects back to whatever IP the webserver sees the request coming from [00:01:46] see /etc/php5/cli/conf.d/20-_remote_debug.ini [00:02:19] tgr: hmm... xdebug.max_nesting_level = "200" [00:02:19] xdebug.remote_connect_back = "1" [00:02:19] xdebug.remote_enable = "1" [00:02:27] but no connections to ide... [00:02:45] 6Labs, 6Operations: revise/fix labstore replicate backup jobs - https://phabricator.wikimedia.org/T127567#2113829 (10chasemp) Backups were failing again last night, and I'm pretty sure it was related to a full snapshot left behind on labstore2001 (which was one of the previous causes). I removed the offending... [00:02:50] when I load e.g. http://cirrustest.wiki.local.wmftest.net:8080/w/api.php?action=cirrus-suggest-index [00:02:51] are you debugging a browser request? [00:03:13] tgr: doesn't matter, it doesn't work both with browser requests and others... [00:03:44] with CLI that's normal, there is an xdebug_on command for that [00:04:54] tgr: what if CLI tool does http request? [00:05:04] e.g. browser tests [00:05:24] check /var/log/xdebug.log (although not 100% sure that's turned on by default), it should say something along the lines of "Remote address found, connecting to 10.0.2.2:9000" [00:05:53] I don't think browser run from CLI will work out of the box [00:06:17] in that case, you would have to set up a port forward [00:07:10] well, if you are running the test in the host machine it's the right IP and it will work [00:07:56] if you are running it on vagrant, you will need to netcat localhost:9000 to 10.11.12.1:9000 or something along those lines [00:08:18] tgr: ok, I'll try to set up remote host to 10.11.12.1 and see [00:08:40] weird thing it worked couple of days ago and then I restarted the VM and it broke :( no idea what happened [00:09:25] xdebug.log is usually a good start with any kind of problem [00:09:58] there's no such log file [00:10:04] do I need to enable something? [00:10:53] xdebug.remote_log must be set, I don't remember if vagrant does that by default [00:11:16] xdebug.remote_log [00:11:32] xdebug.remote_log="/var/log/xdebug.log" [00:11:40] and it must be writable of course [00:15:20] hmm... log doesn't work but if I set remote_host browser connection now works... CLI browsing still weird though... maybe CLI connects to wrong thing? will dig deeper [00:16:42] hmm... CLI browsing connects to localhost still. weird [00:16:51] CLI should work the same way, except that remote_connect_back doesn't do anything useful [00:16:52] maybe it's reding wrong config file? [00:17:27] it's not CLI PHP, it's CLI tool that does request to http://..../api.php [00:17:43] so php is still running in the apache server [00:18:09] my guess would be that you have remote_connect_back on (which takes precendence), for a browser it tries to connect to 10.0.2.2 (which is what vagrant normally sees) and fails and falls back to remote_host, for CLI it sees localhost as the source [00:18:33] ah, maybe [00:18:50] IDE restarts sometimes help [00:19:20] the IDE not listening properly on :9000 or another xdebug instance hogging it or such [00:19:23] IDE is fine, server just trying to connect to localhost instead of IP [00:20:06] I mean, the IDE can be the cause of xdebug failing to connect [00:20:18] halleluyah! after disabling connect back it finally finds the ide [00:20:37] didn't know that takes precedence over explicit ip... [00:22:17] tgr: thanks, I think I got it working now [01:03:16] 6Labs, 10Tool-Labs, 7Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2113936 (10bd808) I'm kind of thinking this interface should start at the landing page of tools.wmflabs.org. Here are some crazy brainstorming ideas: * OAuth integration with SU... [01:12:40] 6Labs, 10Tool-Labs, 7Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2113960 (10bd808) Kind of related: * {T104917} * {T64837} * {T115650} [01:16:33] 6Labs, 10Tool-Labs, 7Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2113966 (10bd808) I guess another question I'm opening up here is whether or not this should be limited to Tool Labs or more broadly useful for all Labs projects. My "glorious vi... [01:32:33] 6Labs, 10Tool-Labs, 7Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2114003 (10yuvipanda) <3 I personally think this should be limited to Tool Labs - Service Accounts were a mistake because they were overgeneralized despite being used only on t... [01:41:17] 6Labs, 6Operations: revise/fix labstore replicate backup jobs - https://phabricator.wikimedia.org/T127567#2114031 (10yuvipanda) Thank you for the indepth investigation and awesome write up, @chasemp! This all sounds great, but I want to challenge one assumption: > Multi-week historical copies as space allows... [01:48:53] 6Labs, 6Operations, 10wikitech.wikimedia.org: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2114039 (10Dzahn) thanks you. yes, the hostname change was intended but not explicitely mentioned. the dump import script thing sounds like a potential puppet fix? [01:50:24] 10PAWS: Implement a sane way to access mysql replicas from PAWS - https://phabricator.wikimedia.org/T120471#2114040 (10yuvipanda) Turns out I can just use https://www.percona.com/doc/percona-toolkit/2.1/pt-kill.html as query killer instead of writing my own :D [01:52:24] 6Labs, 6Operations, 10wikitech.wikimedia.org: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2114054 (10Krenair) >>! In T126385#2114039, @Dzahn wrote: > the dump import script thing sounds like a potential puppet fix? I suppose so, if puppet were installed. [01:52:53] RECOVERY - Puppet failure on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [01:56:02] (03CR) 10Ricordisamoa: "LAGTM" (034 comments) [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [01:56:17] ok, i'm fixing the shinken message now. it bugged me too often :) [01:56:25] Puppet failure is not OK [01:56:31] the puppet run is OK [01:56:56] every i look twice because "failure" [01:57:00] time [02:00:02] RECOVERY - Puppet failure on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [02:21:06] 6Labs, 10Tool-Labs, 7Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#2065676 (10scfc) My $ 0.02: 1. Tool authors sign up for #Tool-Labs once, and they create one tool every x, with the median user maintaining y tools in total. x and y could be c... [02:36:44] PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [02:37:29] 10PAWS: Implement a sane way to access mysql replicas from PAWS - https://phabricator.wikimedia.org/T120471#2114108 (10yuvipanda) 5Open>3Resolved I have, and now queries are killed (not the connections!) after 30minutes. [02:38:58] PROBLEM - Puppet run on tools-web-static-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:04:19] (03PS66) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [03:07:31] (03CR) 10Ricordisamoa: "PS66 makes EntitySuggester pass 'uselang' to wbsearchentities, otherwise if the user is not logged into the wiki, search results may be se" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [03:22:11] 6Labs, 10Tool-Labs: Tools K8S workers run out of space - https://phabricator.wikimedia.org/T129729#2114159 (10yuvipanda) [03:22:20] 6Labs, 10Tool-Labs: Tools K8S workers run out of space - https://phabricator.wikimedia.org/T129729#2114173 (10yuvipanda) p:5Triage>3High [03:26:48] RECOVERY - Puppet run on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [03:31:42] 6Labs, 10Tool-Labs, 13Patch-For-Review: Tools K8S workers run out of space - https://phabricator.wikimedia.org/T129729#2114192 (10yuvipanda) Am doing this for each host now, manually moving the contents of /var/lib/docker around. [03:40:37] 6Labs, 10Tool-Labs: Investigate kubelet container garbage collection - https://phabricator.wikimedia.org/T129730#2114202 (10yuvipanda) [03:43:27] 6Labs, 10Tool-Labs, 13Patch-For-Review: Tools K8S workers run out of space - https://phabricator.wikimedia.org/T129729#2114218 (10yuvipanda) Done for 1001 and repooled it. I'm running a `docker rm $(docker ps -a -q)` on all other nodes to cleanup the docker containers for now before performing this operation... [04:44:38] 6Labs, 10Tool-Labs, 13Patch-For-Review: Tools K8S workers run out of space - https://phabricator.wikimedia.org/T129729#2114244 (10yuvipanda) 5Open>3Resolved Done for all of them! See T129730 for underlying issue. [07:13:40] (03CR) 10Ricordisamoa: Updated nuclides charts to display using SVG; includes some refactoring into common components (032 comments) [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [10:06:10] (03PS1) 10MarcoAurelio: Modify stewardbot code to support I014d244d [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276943 [10:09:32] (03CR) 10MarcoAurelio: [C: 032] Modify stewardbot code to support I014d244d [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276943 (owner: 10MarcoAurelio) [10:10:07] (03Merged) 10jenkins-bot: Modify stewardbot code to support I014d244d [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276943 (owner: 10MarcoAurelio) [11:47:50] RECOVERY - Free space - all mounts on tools-worker-1001 is OK: OK: All targets OK [14:04:48] 6Labs, 10Tool-Labs, 10pywikibot-core, 13Patch-For-Review: Tool Labs Pywikibot does not work with new shared Pywikibot config files - https://phabricator.wikimedia.org/T129406#2114683 (10jayvdb) 5Open>3Resolved Thanks @zhuyifei1999 for fixing this bug my change created. [14:28:18] 6Labs, 10Tool-Labs, 10pywikibot-core, 13Patch-For-Review: Tool Labs Pywikibot does not work with new shared Pywikibot config files - https://phabricator.wikimedia.org/T129406#2114702 (10jayvdb) [15:24:51] 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2114807 (10Glaisher) [15:37:25] 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2114812 (10scfc) Are you aware of [[https://wikitech.wikimedia.org/wiki/Nova_Resource:Catgraph|Catgraph]]? When it was developed and deployed, I was hoping that all resource and speed problems of catscan* would be gone forever, but it is no... [15:56:18] (03PS1) 10Youni Verciti: Phase 2 category dpt/Vocabulaire/Index [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/276974 [16:12:40] (03PS67) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [16:17:35] (03CR) 10Ricordisamoa: "PS67 uses the $dragging argument instead of this.$dragging in DragHelper.prototype.isValidDropTarget" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [16:27:16] (03PS68) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [16:27:18] (03PS1) 10Youni Verciti: New commit to be review on dev branch [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/276976 [16:29:09] (03CR) 10Ricordisamoa: "PS68 removes the special case for statements from DragHelper.prototype.isValidDropTarget" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [18:05:40] 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2114904 (10Magnus) I am aware. Actually, at this point, the best option for category tree operations might be a blazegraph instance, IMHO. But that's beside the point. Is there a catgraph instance running? The links I clicked on were all dea... [18:32:42] (03CR) 10ArthurPSmith: "Thanks for the comments, will fix and update shortly." (034 comments) [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [18:37:28] (03PS3) 10ArthurPSmith: Updated nuclides charts to display using SVG; includes some refactoring into common components [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 [18:40:14] (03PS4) 10ArthurPSmith: Updated nuclides charts to display using SVG; includes some refactoring into common components [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 [18:42:45] (03CR) 10ArthurPSmith: [C: 031] "Ok, I believe I fixed all the things you mentioned last time, take a look thanks!" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [18:43:20] (03CR) 10Ricordisamoa: Updated nuclides charts to display using SVG; includes some refactoring into common components (031 comment) [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [18:43:38] (03CR) 10Ricordisamoa: [C: 04-1] "One last minor thing ;-)" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [19:04:27] (03PS5) 10ArthurPSmith: Updated nuclides charts to display using SVG; includes some refactoring into common components [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 [19:04:55] (03CR) 10ArthurPSmith: "just uploaded fix for this..." (031 comment) [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [19:06:07] (03CR) 10ArthurPSmith: [C: 031] "hope this works!" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [19:14:37] (03CR) 10Ricordisamoa: [C: 032] Updated nuclides charts to display using SVG; includes some refactoring into common components [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [19:19:37] 6Labs: New project: Petscan - https://phabricator.wikimedia.org/T129749#2114943 (10scfc) You are right, the links don't work (`sylvester.wmflabs.org` doesn't resolve), and directly accessing the instance from within Labs gives: ``` scfc@toolsbeta-vagrant3-scfc:/srv/mediawiki-vagrant$ time curl "http://sylvester... [19:28:24] (03Merged) 10jenkins-bot: Updated nuclides charts to display using SVG; includes some refactoring into common components [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [20:50:19] 6Labs, 6Operations, 10wikitech.wikimedia.org: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2115044 (10Krenair) >>! In T126385#2105742, @Krenair wrote: > This should hopefully fix importing. Revisions from earlier today have appeared on https://wikitech-static.wikimedia.o... [22:46:27] 6Labs, 10Horizon, 13Patch-For-Review: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2115084 (10AlexMonk-WMF) Status update: The basic dynamicproxy - horizon integration was working (until I tried to change it to use the service catalogue as suggest... [23:46:36] 6Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 6Operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#2115139 (10Krenair)