[00:00:25] (03PS2) 10Jean-Frédéric: Extract method extract_elements_from_template_param from update_database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281812 [00:00:27] (03PS2) 10Jean-Frédéric: Add unit tests for ucfirst method [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281811 [00:02:25] (03PS3) 10Jean-Frédéric: Extract method extract_elements_from_template_param from update_database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281812 [00:02:27] (03PS3) 10Jean-Frédéric: Add unit tests for ucfirst method [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281811 [00:02:49] (03CR) 10Jean-Frédéric: "Done" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281811 (owner: 10Jean-Frédéric) [00:03:26] (03CR) 10Jean-Frédéric: "Comments adressed." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281811 (owner: 10Jean-Frédéric) [03:57:47] RECOVERY - Puppet run on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:13:51] 10PAWS: R packages not installing correctly (devtools & curl) in Jupyter - https://phabricator.wikimedia.org/T131657#2186098 (10yuvipanda) I'm fixing this now by just straight up using the upstream packaged versions of R, rather than the ones present in debian. [06:37:35] PROBLEM - Puppet run on tools-devpi-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [06:48:38] (03PS71) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [06:52:34] RECOVERY - Puppet run on tools-devpi-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:00:38] (03CR) 10Ricordisamoa: "PS71 updates jQuery from 2.2.0 to 2.2.3" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [07:10:30] 10PAWS: R packages not installing correctly (devtools & curl) in Jupyter - https://phabricator.wikimedia.org/T131657#2186167 (10yuvipanda) Ok, I've the latest version of R installed now. If you kill your server and start again via the control panel, you should be able to do all of those things now :) [07:29:39] PROBLEM - Puppet run on tools-devpi-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:35:01] (03CR) 10Lokal Profil: [C: 032] Add unit tests for ucfirst method [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281811 (owner: 10Jean-Frédéric) [07:35:39] (03Merged) 10jenkins-bot: Add unit tests for ucfirst method [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281811 (owner: 10Jean-Frédéric) [07:35:41] (03Merged) 10jenkins-bot: Extract method extract_elements_from_template_param from update_database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281812 (owner: 10Jean-Frédéric) [07:46:37] 6Labs, 10PAWS, 10Tool-Labs: Setup a devpi server to help speedup pip installs - https://phabricator.wikimedia.org/T132025#2186180 (10yuvipanda) [09:04:44] (03PS1) 10Lokal Profil: Change all linebreaks from Windows to Unix [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/282115 [09:05:33] (03CR) 10Lokal Profil: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/282115 (owner: 10Lokal Profil) [09:12:26] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: BaGLAMa2 has very long running queries - https://phabricator.wikimedia.org/T131933#2186301 (10Magnus) Trying to fix it now. [09:30:32] PROBLEM - Puppet staleness on tools-bastion-10 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [09:36:27] (03CR) 10Jean-Frédéric: [C: 032] Change all linebreaks from Windows to Unix [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/282115 (owner: 10Lokal Profil) [09:37:19] (03Merged) 10jenkins-bot: Change all linebreaks from Windows to Unix [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/282115 (owner: 10Lokal Profil) [10:11:38] 10Tool-Labs-tools-Other, 6Community-Tech, 7Category, 7Community-Wishlist-Survey: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2186372 (10Johan) [12:55:05] 6Labs, 10Tool-Labs, 10DBA, 7Tracking: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#2186517 (10Magnus) [12:55:07] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: BaGLAMa2 has very long running queries - https://phabricator.wikimedia.org/T131933#2186514 (10Magnus) 5Open>3Resolved a:3Magnus I think I fixed it, please re-open if not! [13:13:20] 6Labs, 10Tool-Labs, 6Operations, 7Icinga: tool labs instance distribution monitoring is broken - https://phabricator.wikimedia.org/T119929#2186527 (10Andrew) a:3Andrew [13:34:51] 6Labs, 10Tool-Labs, 6Operations, 7Icinga: tool labs instance distribution monitoring is broken - https://phabricator.wikimedia.org/T119929#2186543 (10Andrew) This script uses the nova-tools-bot credentials, and they are currently invalid. [13:40:55] 6Labs, 10Tool-Labs, 6Operations, 7Icinga: tool labs instance distribution monitoring is broken - https://phabricator.wikimedia.org/T119929#2186548 (10Andrew) Best I can tell, that account has a new password and private hiera has fallen behind. I'm not sure how we would've gotten here though. [13:53:10] PROBLEM - Puppet run on tools-exec-1203 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [13:53:12] PROBLEM - Puppet run on tools-docker-registry-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:53:56] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:54:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:55:53] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:56:23] PROBLEM - Puppet run on tools-worker-1005 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:56:23] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:56:47] PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:56:57] PROBLEM - Puppet run on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:57:03] PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:57:31] PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:57:35] PROBLEM - Puppet run on tools-exec-1205 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:58:01] o.O [13:58:14] PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:58:24] PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:58:40] PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:08:06] YuviPanda: Are you currently here? [14:12:15] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:15:39] RECOVERY - Puppet staleness on tools-bastion-10 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:15:45] I ran on tools-webgrid-lighttpd-1204 directly and it was fine [14:16:06] not sure why teh drift but it seems not be a puppet code issue, the master may have a brief problem [14:22:37] RECOVERY - Puppet run on tools-exec-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:15] its a valhallasw`cloud !? [14:27:17] :P [14:28:34] PROBLEM - Puppet run on tools-exec-1205 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:30:44] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [14:31:58] RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [14:32:12] addshore: I'm innocent! [14:32:16] haha :P [14:32:23] not all labs exec hosts has php5.5? [14:32:34] addshore: precise vs trusty [14:32:35] !log ores deployed ores-wikimedia-config:34ba0fa [14:32:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [14:33:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [14:33:13] RECOVERY - Puppet run on tools-exec-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [14:33:13] RECOVERY - Puppet run on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:33:15] RECOVERY - Puppet run on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [14:33:18] ahh okay! yeh, forgot! must specify trusty! [14:33:48] chasemp: ^[[1;31mError: Could not retrieve catalog from remote server: Error 400 on SERVER: Reading data from Hosts/tools-exec-1405 failed: IOError: Could not correctly fetch revision for Hosts/tools-exec-1405, HTTP status code 500 at /etc/puppet/manifests/realm.pp:11 on node tools-exec-1405.tools.eqiad.wmflabs^[[0m [14:33:57] RECOVERY - Puppet run on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [14:34:13] maybe a wikitech burp so Hiera failed? [14:34:29] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [14:34:34] that seems most probable valhallasw`cloud yeah (I know wikitech did have a few minutes of downtime) [14:34:47] and it roughly coincides [14:36:22] RECOVERY - Puppet run on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [14:36:22] RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [14:36:56] RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [14:43:39] RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [14:45:27] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: Throttling Cyberbot tool user as it is consuming most of the CPU - https://phabricator.wikimedia.org/T131937#2186931 (10Cyberpower678) @Volans since I have now reduced the workers to 2 can the throttle be removed again. You have my word that I won't rest... [15:17:54] chasemp: is it possible to grow the root volume on an instance rather than mounting the spare space as another volume? [15:18:55] subbu has a couple of medium size vms that are filling up but I don't think that mounting the extra 20G would make a lot of difference since all the files are in /srv/... [15:26:35] I know we can do it functionally as in I've done before but I don't think we do it as any kind of existing process, not a punt :) but if you wanted to poke at ondisk resize of a vm volume I could give it whirl on his vm? [15:26:59] from past life it's basically, stop vm, extend image size, extend filesystm size, start vm [15:27:49] cool. let me check with him to see if he's at a good stopping place [15:29:00] I don't k now how that may play into nova compute, andrewbogott any fallout from upping a vm on disk do you think? [15:36:17] We don't support resizing at the moment. Nova claims to do it but I've never seen it work [15:36:32] If horizon claims to do it I should disable the option [15:38:47] bd808: ^ [15:39:21] andrewbogott: *nod* so we should just build larger vms and mount the spare disk then I guess [15:39:39] when you say 'mount the spare disk'... [15:39:51] role::labs::lvm::srv [15:39:54] Ah, yeah [15:40:02] sadly that's the only option at the moment, yeah [15:40:12] what is that role doing at instances? [15:40:55] each VM size has a disk quota (20G, 40G, 60G?, ...) [15:41:09] but the base image will always have a 20G disk [15:41:31] that role will make a volume from the remaining disk in quota and mount it at /srv [15:41:57] so when you do `df` you will see the second mount [15:42:37] 6Labs: Create labs project arcanist - https://phabricator.wikimedia.org/T130507#2187056 (10Luke081515) >>! In T130507#2137956, @bd808 wrote: > Storing user credentials in a Labs instance is a really really bad idea. This would allow anyone who is admin in that project to impersonate anyone who setup credentials... [15:47:50] 6Labs: Create labs project arcanist - https://phabricator.wikimedia.org/T130507#2187109 (10Paladox) @bd808 Maybe we can do http://www.gossamer-threads.com/lists/openssh/users/45984 [15:49:51] 6Labs: Create labs project arcanist - https://phabricator.wikimedia.org/T130507#2187167 (10bd808) >>! In T130507#2187056, @Luke081515 wrote: >>>! In T130507#2137956, @bd808 wrote: >> Storing user credentials in a Labs instance is a really really bad idea. This would allow anyone who is admin in that project to i... [15:54:31] 6Labs: Create labs project arcanist - https://phabricator.wikimedia.org/T130507#2187191 (10Paladox) @bd808 no but if a user doesn't store there key on the server then no one can view the key. They can use ssh-agent as I do which is secure and doesn't show the server my key because it wont be on the server. [15:57:32] 6Labs: Create labs project arcanist - https://phabricator.wikimedia.org/T130507#2187209 (10bd808) I'm still not sure what the problem being solved here actually is. If running `arc` on a Windows host is problematic it seems to me that a local VM based solution (like MediaWiki-Vagrant or something more targeted a... [15:59:19] bd808: http://linux.die.net/man/8/resize2fs suggests expanding should be possible on-line [15:59:51] (I read your question as 'we have a 40GB vm and want to extend / to be 40GB') [16:00:06] valhallasw`cloud: yeah. that's what we'd like to do [16:00:26] I think lvextend + resize2fs should be able to do that ... see http://www.linuxuser.co.uk/features/resize-your-disks-on-the-fly-with-lvm [16:00:36] but I have never tried it [16:00:49] 6Labs: Create labs project arcanist - https://phabricator.wikimedia.org/T130507#2187217 (10Paladox) @bd808 I created this windows installer https://github.com/paladox/Arcanist-installer-for-windows which now works on windows. But maybe something like MediaWiki-Vagrant would work. [16:01:23] we can give it a shot before building new VMs. worst case is that we brick the VM that we were going to replace anyway [16:02:40] oh... but the root volume isn't LVS [16:03:40] hmmm [16:04:12] so if we had mounted a 10G LVS disk we could grow it to 20G [16:04:47] but otherwise I think we are stuck without support at the nova/horizon level [16:05:21] hrm. [16:08:43] bd808: http://possiblelossofprecision.net/?p=228 is not fully online but only a single reboot, so if you're OK with potentially losing the VM, that's something you can try [16:13:15] valhallasw`cloud: I don't think that will work either. The vdisk that openstack presents to the VM is 20G, not 40G with 20G unpartitioned. The magic needs to happen on the openstack side [16:13:50] http://docs.openstack.org/user-guide/common/cli_manage_volumes.html#resize-a-volume [16:14:11] Oh, there are sda for / and sdb for lvm? [16:15:10] but if there is no UI for resizing then I think we are best to just punt and build different VMs. Fancy one-off tricks won't scale [16:21:07] bd808: this is a long-running point of pain… we need some kind of block-storage solution that isn't bound to instance storage (or the virt servers which are super expensive per gb) [16:21:28] There are plenty of official solutions for this but we don't have a clear vision for how to do it yet [16:25:39] andrewbogott: *nod* [16:25:57] just don't say "gluster" xD [16:27:35] 10PAWS: Add a link to this phab board in PAWS - https://phabricator.wikimedia.org/T132069#2187302 (10Halfak) [16:27:55] 10PAWS: Add "less" and "wget" to docker image - https://phabricator.wikimedia.org/T132070#2187316 (10Halfak) [16:43:02] Luke081515: nope :) also in general, just ask - other people are usually smarter than me [16:43:59] YuviPanda: I want to test wikibugs, and there is: "IRCNOTIFIER_KEY": "(ask yuvipanda)", [16:44:06] aaaah [16:44:12] Luke081515: let me PM you a key [16:44:18] ok [16:48:41] bd808: as far as I can see, /dev/vda is the full disk (20, 40, 80 or 160GB), and /dev/vda1 is the root partition [16:48:53] bd808: so I think you should be able to grow /dev/vda1 to be the entire drive [16:50:32] hmm... looks like you are probably right. I guess I was looking at /dev/vda1 rather than /dev/vda [16:53:16] the layout is root, swap, lvm. Do to resize I'd need to drop the swap and lvm, grow the root, add back a swap, reboot, and resize2fs to expand the fs for root [16:53:29] s/do to/to do/ [16:53:55] Yes, that sounds about right [16:54:15] valhallasw`cloud: I got to questions: I want to setup a wikibugs2 bot, to test something, and my open questions are: a) which redis queue name, b) Which script I have to execute to start the IRC-Bot? [16:54:41] Luke081515: something else than the normal wikibugs uses, and redis2irc.py [16:54:42] valhallasw`cloud: I'll try it somewhere after my quarterly review meeting and see if it actually works [16:55:34] valhallasw`cloud: If I'm only using the IRC bot, can I remove the mediawiki user data? (I don't want to write anything at mw.org) [16:55:43] yes [16:55:48] mediawiki is only for the project list export [16:56:52] ok [16:57:33] valhallasw`cloud: I'm normaly executing php files. Can I execute phython files like: $ ./redis2irc.py ? [16:57:57] I think you need to run it in a virtualenv [16:58:12] and then run python redis2irc.py [17:00:46] valhallasw`cloud: sry, maybe the question is obvious and I don't see it, but how to get into /create the virtualenv? [17:01:25] http://docs.python-guide.org/en/latest/dev/virtualenvs/ [17:01:28] virtualenv [17:01:32] source /bin/activate [17:01:36] pip install -r requirements.txt [17:01:42] python redis2irc.py [17:02:45] and can I submit the script to the grid from the virtualenv directly later? [17:05:03] submit /path/to/venv/bin/python /path/to/redis2irc.py [17:05:11] ok, thank you very much [17:05:11] (see fabfile.py) [17:07:45] valhallasw`cloud: I got one error :-/ at the step: pip install -r requirements.txt. Before that, I get the right path, but now I get: [17:07:52] Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt' [17:07:56] Storing debug log for failure in /data/project/luke081515bot-dev/.pip/pip.log [17:08:16] pass the right path to requirements.txt? it's in the git checkout [17:11:07] the data from wikibugs is at a patch "wikibugs2", and after the source command I got a (wikibugs2) at the command line, and the file is in there. [17:12:15] valhallasw`cloud: ^ [17:12:34] Luke081515: ??? [17:12:51] the (wikibugs2) refers to the virtualenv [17:12:58] but you are apparently still in ~ [17:13:02] rather than in ~/wikibugs2? [17:13:30] sorry, my fault [17:18:48] valhallasw`cloud: If I'm running python redis2irc.py the console throws an syntaxerror: [17:18:53] File "redis2irc.py", line 107 [17:18:54] yield from redislistener(bot) [17:18:54] ^ [17:18:54] SyntaxError: invalid syntax [17:19:00] oh, right. [17:19:09] It needs python 3.4+ (3.5+? not sure) [17:20:53] PROBLEM - Host tools-worker-1011 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:11] valhallasw`cloud: How I can solve that? Actually I'm on labs-bastion-0 [17:22:13] 3 [17:22:35] Luke081515: create a virtualenv with a python 3.4 interpreter (see the virtualenv docs) [17:22:50] ok [17:25:15] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:27] valhallasw`cloud: I installed 3.4 now, and get the same error :-/ but if I try 3.5: The executable /usr/bin/python3.5 (from --python=/usr/bin/python3.5) does not exist [17:43:23] valhallasw`cloud: manually resizing the partitions seems to have worked! Interestingly the resize2fs wasn't needed. [17:43:38] I'll add a wiki page about what I did [17:46:16] bd808: what did you end up doing? [17:47:56] chasemp: https://wikitech.wikimedia.org/wiki/Help:Resize_root_partition [17:48:18] Luke081515: deactivate & re-activate the virtualenv? [17:49:56] ok, I will try it [17:52:03] chasemp: so basically I deleted the partition map, created a new one that gave more space to the root partition and then rebooted. [17:52:20] bd808: yeah cool well played [17:52:37] when It came back up I expected to need to run resizefs but when I did it said that the fs was already full size [18:57:46] grr... horizon tells me 'invalid credentials' when attempting to log in, but I set up 2fa and double/tripple-checked my username/password. is there any way to debug it further? [18:59:00] twentyafterfour: is your phone time (and timezone) set correctly? [18:59:37] and you're using 20after4 as login name, right, not twentyafterfour [19:02:00] valhallasw`cloud: yeah [19:03:30] valhallasw`cloud: nevermind, I got it (reset 2fa on wikitech) [19:44:00] 10PAWS: R packages not installing correctly (devtools & curl) in Jupyter - https://phabricator.wikimedia.org/T131657#2187942 (10Halfak) Looks like this is *done*. Please reopen if it is still an issue. [19:44:06] 10PAWS: R packages not installing correctly (devtools & curl) in Jupyter - https://phabricator.wikimedia.org/T131657#2187943 (10Halfak) 5Open>3Resolved [20:16:37] RECOVERY - Puppet run on tools-exec-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [20:31:09] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: Throttling Cyberbot tool user as it is consuming most of the CPU - https://phabricator.wikimedia.org/T131937#2188130 (10Volans) @Cyberpower678, thanks for the collaboration, I've removed the limit for now: ``` GRANT USAGE ON *.* TO 's51059' WITH MAX_USER... [20:53:50] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: BaGLAMa2 has very long running queries - https://phabricator.wikimedia.org/T131933#2188219 (10Volans) @Magnus: thanks for the collaboration and quick fix, I don't see those queries for now, I'll let you know in case we see them again. [20:55:10] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Lolsa112 was created, changed by Lolsa112 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Lolsa112 edit summary: Created page with "{{Tools Access Request |Justification=I will be working on the wakt-beta tool, which is an application that allows users to geotag public artwork in Sweden. WAKT is a project..." [21:03:06] 10Tool-Labs-tools-wikiloves: Desenvolver gráficos e tabelas para exibir os dados - https://phabricator.wikimedia.org/T131192#2188228 (10Danilo) Adicionei os gráficos, vejam o que acham e se precisa de alguma correção ou modificação. http://tools.wmflabs.org/wikiloves/ [21:15:48] 6Labs, 10Tool-Labs, 6Operations, 7Icinga: tool labs instance distribution monitoring is broken - https://phabricator.wikimedia.org/T119929#2188266 (10Andrew) The username/password issue is resolved on palladium; a fix for another bug is attached. After all that, the test will still fail because the bastio... [22:48:05] legoktm: Are you currently here? [23:03:26] RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0]