[00:00:22] ssh to labs has always worked [00:00:36] cool, just checking [00:00:43] since it’s different keys in different places [00:04:01] * andrewbogott is out for now [00:04:43] should this work now? [00:04:45] ssh -v -p 29418 hroest@gerrit.wikimedia.org [00:05:46] hmm it doesnt [00:07:32] $ ssh -v -p 29418 hroest@gerrit.wikimedia.org [00:07:49] debug1: Authentications that can continue: publickey [00:07:49] debug1: Next authentication method: publickey [00:07:50] debug1: Offering RSA public key: /home/hr/.ssh/id_rsa [00:07:50] debug1: Authentications that can continue: publickey [00:07:50] debug1: Trying private key: /home/hr/.ssh/id_dsa [00:07:50] debug1: Trying private key: /home/hr/.ssh/id_ecdsa [00:07:52] debug1: No more authentication methods to try. [00:07:54] Permission denied (publickey). [00:11:32] maybe there is another key? [00:54:05] hannes_roest, still around? [00:54:51] hannes_roest, https://gerrit.wikimedia.org/r/#/settings/ssh-keys [00:54:56] separate keys [00:55:02] there's an upstream task somewhere [01:31:59] Krenair, [01:32:00] yes still here [01:33:20] ssh gerrit [01:33:20] **** Welcome to Gerrit Code Review **** [01:33:20] Hi hroest, you have successfully connected over SSH. [01:33:20] Unfortunately, interactive shells are disabled. [01:33:23] great, i think it worked [01:36:42] yep [01:41:58] Krenair, is there a way to disable the email check? [01:42:14] or should I just registe ranother email? [01:42:48] the email *check*? [01:42:57] sorry, not sure what you're referring to [01:43:16] it seems that "git review" checks my email [01:43:32] remote: ERROR: committer email address xxx [01:43:32] remote: ERROR: does not match your user account. [01:43:39] but I think I can just register another email [01:43:46] oh, you should register the email address you want to commit with under your gerrit account [01:43:52] RECOVERY - Puppet staleness on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [3600.0] [01:44:49] ok, I did [02:05:36] 6Labs, 10Tool-Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org, and 2 others: Create a wiki documentation page for each tool - https://phabricator.wikimedia.org/T122865#2053930 (10bd808) [02:08:31] 6Labs, 6Developer-Relations, 10wikitech.wikimedia.org, 7Epic: [EPIC] Make wikitech more friendly for the multiple audiences it supports - https://phabricator.wikimedia.org/T123425#2053933 (10bd808) [02:08:36] 6Labs, 10Tool-Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org, and 2 others: Create a wiki documentation page for each tool - https://phabricator.wikimedia.org/T122865#2053931 (10bd808) 5Open>3Resolved The Tool namespace is live on wikitech. I have also moved the tool documentation pages that... [02:20:48] 6Labs, 10wikitech.wikimedia.org: Exclude nova resource pages from *default* wikitech search - https://phabricator.wikimedia.org/T122993#2053960 (10bd808) Search results are looking pretty good now that the deboosting is in place and the bug that @EBernhardson found with multiple content namespaces has been fix... [04:01:55] 6Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure: Make labs wikitech role aware - https://phabricator.wikimedia.org/T127771#2053464 (10scfc) Some Labs projects use the `classes` Hiera variable set for example at https://wikitech.wikimedia.org/wiki/Hiera:Tools to apply (role) classes to all instanc... [04:12:17] 6Labs, 10Tool-Labs: setup-tomcat does not work - https://phabricator.wikimedia.org/T118094#1791466 (10zhuyifei1999) FYI: https://gerrit.wikimedia.org/r/#/c/130429/2/misctools/setup-tomcat Maybe we can track down this file. [04:51:53] 6Labs, 10Tool-Labs: setup-tomcat does not work - https://phabricator.wikimedia.org/T118094#2054125 (10zhuyifei1999) It's still here: https://github.com/wikimedia/labs-toollabs/blob/master/misctools/setup-tomcat, but somehow it isn't included in Makefile.am, nor there's a manpage for it. [05:33:27] 6Labs, 6Collaboration-Team-Backlog, 10Flow, 10wikitech.wikimedia.org: Enable Flow for Tool namespace on wikitech - https://phabricator.wikimedia.org/T127792#2054159 (10bd808) [05:33:44] 6Labs, 6Collaboration-Team-Backlog, 10Flow, 10wikitech.wikimedia.org: Enable Flow for Tool namespace on wikitech - https://phabricator.wikimedia.org/T127792#2054173 (10bd808) [05:33:46] 6Labs, 6Developer-Relations, 10wikitech.wikimedia.org, 7Epic: [EPIC] Make wikitech more friendly for the multiple audiences it supports - https://phabricator.wikimedia.org/T123425#2054172 (10bd808) [06:16:25] 6Labs, 10Tool-Labs, 6Collaboration-Team-Backlog, 10Flow, 10wikitech.wikimedia.org: Enable Flow for Tool namespace on wikitech - https://phabricator.wikimedia.org/T127792#2054198 (10bd808) [06:30:31] 6Labs, 10Tool-Labs, 10pywikibot-core: pywikibot broken on wikimedia labs - https://phabricator.wikimedia.org/T126666#2020257 (10bd808) >>! In T126666#2021312, @Ladsgroup wrote: > OK, I think [[https://github.com/pywikibot/Pywikibot-nightly-creator/pull/3|PR]] will fix the issue for now. Feel free to merge it... [06:31:24] 6Labs: Create a "Beginners guide to creating a Labs instance" wiki page - https://phabricator.wikimedia.org/T126094#2054231 (10bd808) [06:31:26] 6Labs, 10Tool-Labs, 7Documentation: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#2054230 (10bd808) [06:58:50] 6Labs, 10Tool-Labs, 7Documentation: Run a documentation sprint for Labs - https://phabricator.wikimedia.org/T101659#1344508 (10Dzahn) Can we add the labs part of T127797 ? [08:08:28] 6Labs, 10Labs-Infrastructure: Labs bandwidth is aleatory/low - https://phabricator.wikimedia.org/T124960#1971067 (10Nemo_bis) I think a traceroute may help here. In my experience the Labs networking is extremely poor in some direction, decent in others. [08:55:24] 6Labs, 10Tool-Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org, and 3 others: Create a wiki documentation page for each tool - https://phabricator.wikimedia.org/T122865#2054458 (10Qgil) [09:12:29] 6Labs, 10Labs-Infrastructure: Make labs wikitech role aware - https://phabricator.wikimedia.org/T127771#2054496 (10hashar) [09:34:24] 6Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2054540 (10jcrespo) Just chiming it to offer some help if moving pdns database around (which is really easy) is needed, and reminding to give some love to T126251, which may be semi-related? [10:48:26] (03PS1) 10Youni Verciti: Fix the .gitignore file & remove *.pyc once more. [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/272697 [10:50:52] (03CR) 10Youni Verciti: "Learning Git/gerrit." [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/272697 (owner: 10Youni Verciti) [11:09:11] 6Labs, 10Tool-Labs, 10DBA: Tool Labs queries die - https://phabricator.wikimedia.org/T127266#2037644 (10jcrespo) 5 hour queries are not allowed on replica servers. The workaround is to split the query in smaller ones. I temporarily restricted queries to 1 hour until I discover the tool responsible of slowne... [11:09:31] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 10DBA: tools.ptwikis throttled for excessive usage of labs db replica resources - https://phabricator.wikimedia.org/T127228#2054816 (10jcrespo) 5Open>3Resolved [11:14:13] 6Labs: Webservice not starting - https://phabricator.wikimedia.org/T127817#2054839 (10Prolineserver) [11:17:21] (03PS1) 10Zhuyifei1999: misctools: add setup-tomcat to Makefile.am [labs/toollabs] - 10https://gerrit.wikimedia.org/r/272699 (https://phabricator.wikimedia.org/T118094) [11:20:12] 6Labs, 10Tool-Labs, 10DBA: Tool Labs queries die - https://phabricator.wikimedia.org/T127266#2054869 (10jcrespo) I should explain that the "5 hour queries" is not something definitive; we are currently with reduced redundancy temporarily due to hardware failure, so I am asking kindly to not overload much the... [11:27:19] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:19] bastion-05 is wading through some thick soup now... sluggish ... [11:28:17] seems to be related to disk? [11:37:02] grrrit-wm: Soory i was off! [11:37:12] -o+r [11:37:53] grrrit-wm is a bot [11:40:26] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 790866 bytes in 8.745 second response time [11:41:07] Stigmj: is it doing better now? [11:41:48] Krenair: Oh thanks! [11:43:13] Krenair: i'm new user of gerrit, hope i'm not doing anythig wrong! [11:46:53] I'm the owner of the tool vocabulary-index, i would be abble to push modifications in it. [11:49:27] But when i push, even if the command return no-error, i can't see modifications in the commit... [11:50:55] are you uploading a new commit or amending an existing change? [11:51:19] godog: seems like it yes. [11:55:19] I probably try both, i mean i can crate new commits ; but if i create a new commit for each changes, i will have soon a lot of commits. [11:56:00] So i was thinking about pushing modifications in an existing commit, but... [12:04:43] i mean but even when the push command seems to run right, i have no result in gerrit i cant see the modifications. [14:00:28] 6Labs, 10Tool-Labs, 13Patch-For-Review: setup-tomcat does not work - https://phabricator.wikimedia.org/T118094#2055285 (10scfc) @Base, @Shoichi: I have manually installed `setup-tomcat` on all instances, so running it should work now. @zhuyifei1999: Thanks for the patch! There's a minor problem with it whi... [14:04:19] (03CR) 10Tim Landscheidt: [C: 04-1] "Thanks for the patch! Two small issues:" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/272699 (https://phabricator.wikimedia.org/T118094) (owner: 10Zhuyifei1999) [15:18:58] 6Labs, 10Labs-Infrastructure: nfs-export alert on labstore1001 - https://phabricator.wikimedia.org/T127835#2055397 (10Andrew) [15:21:23] 6Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2055412 (10Andrew) This just happened again today, /after/ the above firewall patch was applied. So, that wasn't it! Kiall is hopeful that the 'Master=false' element of the above patch will make a signif... [15:29:29] 6Labs, 10Labs-Infrastructure: nfs-export alert on labstore1001 - https://phabricator.wikimedia.org/T127835#2055397 (10scfc) Isn't this one of the services that is required to run exactly on one NFS server? [15:40:48] apergos: what can you tell me about labs instance 'dumps-stats' [15:40:55] oops, not here [16:06:55] 6Labs, 10Labs-Infrastructure: labvirt1006 disk space alert - https://phabricator.wikimedia.org/T127840#2055565 (10Andrew) [16:10:27] Nemo_bis, I would like to migrate dumps-stats to a new virt host, which will cause some downtime. Is that ok? [16:11:58] 6Labs, 10Labs-Infrastructure: labvirt1006 disk space alert - https://phabricator.wikimedia.org/T127840#2055578 (10Andrew) Correction -- dumps-stats is run by nemo and/or hydriz, not Ariel. [16:14:21] 6Labs, 10Tool-Labs, 13Patch-For-Review: setup-tomcat does not work - https://phabricator.wikimedia.org/T118094#2055598 (10Shoichi) @scfc Thank you . I tested. > setup-tomcat Indeed appears,but setup-tomcat won't build ~/public_tomcat/ as what[[ https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#Java_.2... [16:18:47] 6Labs, 10Tool-Labs, 13Patch-For-Review: setup-tomcat does not work - https://phabricator.wikimedia.org/T118094#2055604 (10Shoichi) @scfc Sorry I am wrong, setup-tomcat give me the replying: 
Setting up your public_tomcat directory... Unable to run job: Job was rejected because job requests unknown queu...
[16:18:55] 	 PROBLEM - Puppet failure on tools-exec-1217 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[16:18:55] 	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[16:20:25] 	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[16:20:31] 	 6Labs, 10Labs-Infrastructure: labvirt1006 disk space alert - https://phabricator.wikimedia.org/T127840#2055611 (10Hydriz) Andrew, do you have a maintenance window available for completing this task? Downtime on dumps-stats is fine provided we are given notice on the exact timing.
[16:22:19] 	 PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0]
[16:28:00] 	 6Labs, 10Labs-Infrastructure: labvirt1006 disk space alert - https://phabricator.wikimedia.org/T127840#2055665 (10Andrew) @Hydriz, migrating an instance is slow, and it's difficult to predict the length of the downtime.  At a rough guess, I'd say it will take two to three hours.  If you are able to delete unne...
[16:52:22] 	 RECOVERY - Puppet failure on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:53:52] 	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:55:22] 	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:00:16] 	 6Labs, 10Tool-Labs, 6Collaboration-Team-Backlog, 10Flow, 10wikitech.wikimedia.org: Enable Flow for Tool namespace on wikitech - https://phabricator.wikimedia.org/T127792#2054159 (10Mattflaschen) Are we able to use extension1 here (that's where we normally have flowdb, except private wikis)?  I know wikit...
[17:00:54] 	 6Labs, 10Tool-Labs, 6Collaboration-Team-Backlog, 10Flow, 10wikitech.wikimedia.org: Enable Flow for Tool namespace on wikitech - https://phabricator.wikimedia.org/T127792#2054159 (10Krenair) No, wikitech cannot depend on extension1.
[17:04:07] 	 6Labs, 10Tool-Labs, 6Collaboration-Team-Backlog, 10Flow, 10wikitech.wikimedia.org: Enable Flow for Tool namespace on wikitech - https://phabricator.wikimedia.org/T127792#2055826 (10jcrespo) Flow can be enabled, AFAIK, but it will have to live on its own database (like officewiki or some other one?).
[17:04:37] 	 6Labs, 10Tool-Labs, 6Collaboration-Team-Backlog, 10Flow, 10wikitech.wikimedia.org: Enable Flow for Tool namespace on wikitech - https://phabricator.wikimedia.org/T127792#2055842 (10Mattflaschen)
[17:18:28] 	 6Labs, 10Tool-Labs, 6Collaboration-Team-Backlog, 10Flow, 10wikitech.wikimedia.org: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2055916 (10Mattflaschen)
[17:43:26] 	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:48:25] 	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 790785 bytes in 6.192 second response time
[18:12:25] 	 6Labs, 10Labs-Infrastructure, 6Operations, 10Traffic, 7HTTPS: add a https-only option to dynamicproxy - https://phabricator.wikimedia.org/T120486#2056168 (10Aklapper)
[18:15:24] 	 ^andrew is that you?
[18:15:30] 	 andrewbogott: 
[18:15:52] 	 chasemp: the toollabs home page thing?
[18:16:06] 	 yeah
[18:16:08] 	 I don’t think it’s me, but I may have caused a dns stutter right at the instant of the test
[18:16:15] 	 ok
[18:16:18] 	 (I am restarting things an awful lot right now)
[18:16:31] 	 I checked as soon as the alert fired and it looked fine to me
[18:16:38] 	 me too
[18:16:50] 	 what's the status of the bastions btw?
[18:17:02] 	 01 and 02 are out of commission and 05 is the main
[18:17:05] 	 ?
[18:19:16] 	 ah, sorry
[18:19:24] 	 chasemp: 02 is the same as always
[18:19:26] 	 05 is the new 01
[18:19:31] 	 01 is in ‘shutdown’ state
[18:19:48] 	 (this is tools bastions, of course.  Regular labs bastions are same as always)
[18:19:51] 	 oh ok gotcha, I thought 02 was stale as well sounds good
[18:19:52] 	 right
[19:01:01] 	 bd808: Currently here? I created two pastes at phab with the config from my CA cluster
[19:04:01] 	 hello! Is the tool labs having problems?
[19:04:18] 	 i have one process in qw and one running
[19:11:32] 	 marmick: there aren’t any issues that I know of.
[19:11:39] 	 How long have has your task been pending?
[19:11:52] 	 hm… at least one extra word in that last line
[19:12:12] 	 I was able to submit a job and see it dispatched fwiw
[19:14:15] 	 it was happening 5h ago that tasks were a bit stuck
[19:14:22] 	 the last one stuck is from 20 min ago
[19:15:37] 	 Hi. I keep receiving this mail "Alert:  puppet failed on ee-flow-extra.editor-engagement.eqiad.wmflabs" ut I have no clue of what I am suppposed to do. Any hint?
[19:16:40] 	 Mpaa-irc:  You’re getting that because you’re an admin for the project.  It might be that someone else created and/or broke that instance...
[19:16:51] 	 but in theory as an admin you would know the other admins and know who set it up :)
[19:18:15] 	 Mpaa-irc: I think I see the problem, want me to explain or just fix it myself?
[19:18:21] 	 (it’s an easy one!)
[19:19:01] 	 specifically, I think you’ve run in to this:  https://phabricator.wikimedia.org/T127698
[19:20:43] 	 andrewbogott, if you can teach me how too fix it, glad to learn .... if too long, just fix it :-)
[19:20:58] 	 Mpaa-irc: ok, I think this isn’t too painful...
[19:21:03] 	 you know what puppet is, for starters?
[19:21:20] 	 i guess so ...
[19:22:07] 	 In brief, puppet manages the configuration of a host.  When puppet is stopped or fails, that instance has ‘gone rogue’ and makes the Labs ops nervous.  It could have lingering security problems, logins could break, etc.
[19:22:14] 	 puppet refreshes the config automatically every 20 minutes
[19:22:22] 	 but you can refresh by hand, and get output, by running
[19:22:25] 	 $ sudo puppet agent -tv
[19:22:26] 	 on the host
[19:22:54] 	 So that’s stage one in answering the ‘broken how?’ question
[19:26:31] 	 andrewbogott, hhmm I am not sure I have sudo rights ... it asks for a password ...
[19:26:52] 	 huh, if you’re a project admin but don’t have sudo, that’s weird :)
[19:26:55] * andrewbogott looks
[19:28:07] 	 hm, all project members should have sudo
[19:29:45] 	 Mpaa-irc: maybe this is a chicken-and-egg thing…  I’ll fix puppet and maybe you’ll get sudo back
[19:29:50] 	 although it shouldn’t work like that :(
[19:29:53] 	 andrewbogott, then I do not know where to get the password, as I do not recall I never set one ... I connect via ssh -A mpaa@tools-login.wmflabs.org
[19:30:08] 	 Mpaa-irc: wha?
[19:30:21] 	 the email you’re getting is from ee-flow-extra.editor-engagement.eqiad.wmflabs
[19:30:26] 	 so why ssh to tools-login?
[19:32:58] 	 andrewbogott, ok, you mean when I go to tools.mpaatools@tools-bastion-05, then I just use 'become mpaatools'
[19:33:11] 	 chasemp: any idea how to troubleshoot marmick’s issue?  We’ve pm’d a bit and it seems like he’s doing reasonable things.  
[19:33:46] 	 Mpaa-irc:  Labs consists of hundreds of different servers.
[19:33:54] 	 The alert you’re getting is from the one named ee-flow-extra.editor-engagement.eqiad.wmflabs
[19:34:00] 	 to fix that one you would need to log into that server
[19:34:51] 	 but, don’t worry, I’ll fix it
[19:35:03] 	 if you don’t have anything to do with the editor-engagement project you can just remove yourself as projectadmin.
[19:36:51] 	 andrewbogott, it is geting too painful :-( I have o clue of that server  ... even more that I am an admin there ...
[19:37:03] 	 Mpaa-irc: no worries, I can remove you
[19:37:22] 	 andrewbogott, thanks
[19:37:46] 	 !log editor-engagement removing Mpaa as a projectadmin since they have never heard of this project :)
[19:37:49] 	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Editor-engagement/SAL, dummy
[19:38:22] 	 Mpaa-irc: in any case, thanks for reporting the issue; it’s fixed now
[19:39:54] 	 andrewbogott: and marmick not specifically what's the job id(s)?
[19:40:48] 	 PROBLEM - SSH on tools-webgrid-lighttpd-1206 is CRITICAL: Server answer
[19:47:37] 	 chasemp: well, marmick seemed highly motivated, perhaps they will reappear
[19:48:08] 	 I may be able to poke a bit on that job status but gotta have more details
[19:48:17] 	 andrewbogott, thanks to you, I must have been admin in another life :-)
[19:49:00] 	 chasemp: yeah, I don’t know what tool or job or anything at the moment
[19:50:40] 	 chasemp: 3753766
[19:50:47] 	 RECOVERY - SSH on tools-webgrid-lighttpd-1206 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0)
[19:50:54] 	 chasemp: but it happens with any other task. i can only send one task to the labs
[19:52:02] 	 marmick: you have have specified precise and h_vmem=15360000k
[19:52:17] 	 no box can satisfy that
[19:52:20] * andrewbogott counts zeros
[19:52:20] 	 so it never runs
[19:52:30] 	 ahh
[19:52:33] 	 that’s a lot of zeros!
[19:52:57] 	 i see! 
[19:53:03] 	 how much should i specify?
[19:53:17] 	 i just killed the process
[19:53:51] 	 marmick: there is a default mem reservation, what commadn are you running to schedule?
[19:54:23] 	 jsub -mem X name.sh
[19:54:30] 	 where X is the memory in MB
[19:56:31] 	 well, 14Gigs is probably outside of our arena, you could try 1Gig and see how it goes, I don't known where the original value comes from
[19:56:45] 	 ok
[20:02:00] 	 marmick: drop me a line when you get through your work I'm really curious to see what you are coming up with :)
[20:02:37] 	 :))) chasemp i will! 
[20:02:39] 	 thanks
[20:03:40] * andrewbogott is out for a bit
[20:49:29] 	 does anyone knows why I got this message from the database? 
[20:49:31] 	 "ERROR 1030 (HY000) at line 1: Got error 1152 "Unknown error 1152" from storage engine TokuDB "
[20:57:04] 	 bd808: Are you here?
[20:57:38] 	 Luke081515: yeah. haven't had time to look at your pastes yet
[20:57:53] 	 ok
[20:58:04] 	 don't hurry ;)
[20:58:56] 	 marmick: code code 1152 is ER_ABORTING_CONNECTION
[20:59:23] 	 so it's the mysql server droping my query
[20:59:28] 	 Platonides: thx
[20:59:29] 	 Luke081515: it will probably be a while. I'm doing "real work" at work today :)
[20:59:40] 	 np ;)
[21:00:12] 	 6Labs, 10Tool-Labs, 10DBA: Tool Labs queries die - https://phabricator.wikimedia.org/T127266#2057121 (10Dispenser) The query-killer daemon (IIRC) was written in Java and open source in the DaB's Toolserver SVN repo.  It recognized flags: * `/* SLOW_OK */` Indicates query is not a runaway or butchered query *...
[22:24:05] 	 10Wikibugs, 10Differential, 5Gerrit-Migration: Broadcast Differential activity to IRC - https://phabricator.wikimedia.org/T116330#2057600 (10greg) 5Open>3stalled
[22:40:20] 	 andrewbogott: yes, dumps-stats is ok to shutdown
[22:41:19] 	 Nemo_bis: Hydriz requested a warning window… https://phabricator.wikimedia.org/T127840
[22:52:49] 	 andrewbogott: the instance is unused in this very moment, AFAIK he already interrupted his jobs
[22:53:19] 	 ah, great, I’ll start the migrate right now then.  Thank you!
[22:53:49] 	 Nemo_bis: can you comment on that bug accordingly?
[22:55:06] 	 6Labs, 10Labs-Infrastructure: labvirt1006 disk space alert - https://phabricator.wikimedia.org/T127840#2057712 (10Nemo_bis) I've freed 16 GB with two temporary files of mine, the rest is negligible.
[22:55:13] 	 Already left a note
[22:55:19] 	 thanks!
[22:57:15] 	 6Labs, 10Labs-Infrastructure: labvirt1006 disk space alert - https://phabricator.wikimedia.org/T127840#2057722 (10Nemo_bis) As for tasks, I see Hydriz isn't running anything (apart from a non-urgent python script which is currently idle) and I'm not either, so all fine.
[22:58:00] 	 !log dumps migrating dumps-stats to labvirt1010
[22:58:04] 	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Dumps/SAL, dummy
[23:05:15] 	 andrewbogott: irc just wigged out on me fyi, I'm around but getting some turbulence
[23:17:01] 	 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#1914577 (10Yurik) Graph:PageViews template is now done, and can be inserted anywhere to show the pageviews of any wiki page on any site. For a demo, see [[ https...
[23:30:19] 	 RECOVERY - Puppet staleness on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [3600.0]