[00:06:42] (03PS1) 10Jean-Frédéric: Port categorize_images.py to core using compat2core [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233337 [00:06:59] (03CR) 10Jean-Frédéric: [C: 032 V: 032] Port categorize_images.py to core using compat2core [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233337 (owner: 10Jean-Frédéric) [01:44:49] (03PS1) 10Jean-Frédéric: Fixes to categorize-images [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233339 [01:45:04] (03CR) 10Jean-Frédéric: [C: 032 V: 032] Fixes to categorize-images [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233339 (owner: 10Jean-Frédéric) [02:23:24] (03PS1) 10Jean-Frédéric: Support for Wikidata while guessing categories from CommonsCat [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233340 [02:30:42] (03PS2) 10Jean-Frédéric: Support for Wikidata while guessing categories from CommonsCat [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233340 (https://phabricator.wikimedia.org/T110003) [07:21:35] Change on 12www.mediawiki.org a page Developer access was modified, changed by 171.254.36.74 link https://www.mediawiki.org/w/index.php?diff=1859313 edit summary: [-1541] Blanked the page [07:22:09] Change on 12www.mediawiki.org a page Developer access was modified, changed by Valhallasw link https://www.mediawiki.org/w/index.php?diff=1859314 edit summary: [+1541] Undo revision 1859313 by [[Special:Contributions/171.254.36.74|171.254.36.74]] ([[User talk:171.254.36.74|talk]]) [07:52:55] 10Tool-Labs-tools-Erwin's-tools: mysql library deprecated - https://phabricator.wikimedia.org/T109591#1566412 (10Nemo_bis) Well that's bizarre, what changed on the hosts? I don't see any upgrade on https://lists.wikimedia.org/pipermail/labs-announce/2015-August/ [07:56:43] 10Tool-Labs-tools-Erwin's-tools: mysql library deprecated - https://phabricator.wikimedia.org/T109591#1566417 (10valhallasw) Trusty hosts have been the default webservice host since april: http://thread.gmane.org/gmane.org.wikimedia.labs.announce/11/focus=3628 [08:21:44] 10Tool-Labs-tools-Erwin's-tools: mysql library deprecated - https://phabricator.wikimedia.org/T109591#1566481 (10Nemo_bis) But this error came up just these days. [08:21:55] hi. http://toolserver.org/~kolossos/openlayers/kml-on-ol.php is not working. is it possible to restart the service. the tool maintainer seems to be busy [08:22:30] i just notice it's a toolserver.org link... but it used to work until some days ago [08:22:47] is there a replacement on labs? [08:25:11] 10Tool-Labs-tools-Erwin's-tools: mysql library deprecated - https://phabricator.wikimedia.org/T109591#1566490 (10valhallasw) > Starting April 29 2015, any **new** lighttpd webservices started will default to Ubuntu trusty instead of ubuntu precise as is the case now. Note that none of the current webservices wil... [08:25:16] Greutiste: the toolserver.org server is down because of a security issue [08:25:24] Greutiste: it's probably kmlexport? [08:25:31] https://tools.wmflabs.org/kmlexport/ [08:25:35] the redirect is down, yeah. [08:27:09] no, kmlexport is a different tool kml-on-ol is used on https://meta.wikimedia.org/wiki/Template:GeoTemplate/osm#Wikipedia [08:27:35] Greutiste: the third link in that list is https://tools.wmflabs.org/wiwosm/osm-on-ol/commons-on-osm.php?lat={latdegdec}&lon={londegdec}&zoom={osmzoom} [08:27:38] so that, then? [08:35:43] tool replaced: https://meta.wikimedia.org/w/index.php?title=Template:GeoTemplate/osm&diff=13288755&oldid=12860062 . seems to work [08:36:10] Greutiste: thanks! [09:09:25] 6Labs, 6operations: labstore1002 out of space in vg to create new snapshots - https://phabricator.wikimedia.org/T109954#1566600 (10yuvipanda) All better now, once the lockdirs were deleted. Not sure what the original cause of failure or the cause of the cascade was. [09:12:21] 6Labs, 6operations: labstore1002 out of space in vg to create new snapshots - https://phabricator.wikimedia.org/T109954#1566601 (10yuvipanda) we need to tighten up monitoring, and also provide actual documentation for how to recover from one. I've written up some notes at https://wikitech.wikimedia.org/wiki/NF... [09:40:49] sitic: wanna plan to go to the WMDE office sometime next week? :) [10:12:44] 6Labs, 10Datasets-General-or-Unknown, 10Labs-Infrastructure, 10Wikidata: Wikidata JSON entity dumps not being copied correctly on labs - https://phabricator.wikimedia.org/T109830#1566803 (10Lydia_Pintscher) Sounds related to T107226. [10:38:11] Hey guys, one of my tools have PHP Version 5.3 and one 5.5, I don't know what made the difference, can I use used 5.5 for first also? [10:54:42] The question simply is, how I can use newer php on labs? [11:01:28] ebraminio: hey [11:01:35] ebraminio: webservice --release trusty start [11:01:39] should give you 5.5 [11:01:44] it's also the default for new webservices [11:03:39] YuviPanda: Thank you :) Great [11:12:37] YuviPanda: https://phabricator.wikimedia.org/T110022 ! [11:12:41] #wants [11:13:23] :P [11:13:27] you can't con me into writing PHP! [11:13:41] bah, but I have no time :P [11:13:58] at least not today :D [11:14:48] maybe I will stick around with a beer later and see where I get too.... [11:17:42] addshore: :D [11:17:43] doit [11:42:08] 6Labs, 10Tool-Labs, 6Engineering-Community, 6WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#1566960 (10Qgil) p:5Normal>3Low [11:47:49] 6Labs, 10Tool-Labs: Create a fonts CDN for use on Tool Labs - https://phabricator.wikimedia.org/T110027#1566983 (10Ricordisamoa) 3NEW [11:49:44] 6Labs, 10Tool-Labs: Create a fonts CDN for use on Tool Labs - https://phabricator.wikimedia.org/T110027#1567001 (10yuvipanda) We basically need to run a copy of http://fontcdn.org/ [11:52:28] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries and fonts - https://phabricator.wikimedia.org/T103934#1567010 (10Ricordisamoa) [11:53:09] 6Labs, 10Tool-Labs: Create a fonts CDN for use on Tool Labs - https://phabricator.wikimedia.org/T110027#1567014 (10Ricordisamoa) [11:53:09] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries and fonts - https://phabricator.wikimedia.org/T103934#1403141 (10Ricordisamoa) [11:53:49] 6Labs, 10Tool-Labs: Create a fonts CDN for use on Tool Labs - https://phabricator.wikimedia.org/T110027#1567015 (10yuvipanda) With a caching proxy back to Google CDN even maybe, so we get all the features. [13:28:10] 6Labs, 6operations: labs salt master on jessie fails to install salt-master - https://phabricator.wikimedia.org/T110032#1567209 (10fgiunchedi) 3NEW [13:30:56] 6Labs, 6operations: labs salt master on jessie fails to install salt-master - https://phabricator.wikimedia.org/T110032#1567224 (10fgiunchedi) [13:39:41] 6Labs, 6operations: labs salt master on jessie fails to install salt-master - https://phabricator.wikimedia.org/T110032#1567257 (10fgiunchedi) it seems due to a version mismatch from what gets installed on the image by default (since `grep salt-minion /var/log/dpkg.log*` yields no results) ```lines=15 filippo... [13:41:28] thoughts where I could find more info about ^ ? [13:42:00] godog: might be part of the image itself, I guess? [13:42:11] not sure where to look for it [13:44:00] godog: it's not clear to me from that apt log why it errors out, though? I.e. why it doesn't just install the 2014 version. [13:44:29] maybe because it needs explicit permission to downgrade? not sure what the -y vs --force-yes thing is [13:44:36] yeah I think that's it valhallasw`cloud [13:44:51] force the downgrade, which doesn't happen with -y alone [13:45:47] YuviPanda: yup I think so too part of the image, I'll ask andrewbogot.t [13:46:11] or actually andrewbogott ^ [13:47:14] so puppet only passes --force-yes if you do force => '2014.7.5+ds-1' [13:47:17] eh, [13:47:24] ensure => '2014..etc' [13:47:29] https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/package/apt.rb#L65 [13:47:45] godog: I can look — but be warned that changing the base image doesn’t happen frequently, so even if that’s the right change it’s unlikely to be made available to you right away [13:47:55] Unless it’s for some reason urgent or a labs-wide issue [13:48:38] heh, some (all?) jessie instances won't have role::salt::masters::labs::project_master work out of the box [13:48:56] not sure if that qualifies as urgent, certainly jessie-wide [13:49:11] it's also fixable on the puppet level, though [13:49:11] isn’t running salt master on jessie weird? We don’t do that in prod at the moment do we? [13:49:27] by allowing the downgrade (or by forcing the 2015 version) [13:49:37] + there’s a by-hand workaround, right? [13:49:53] yup both correct [13:49:58] valhallasw`cloud: (several days later): does this look right to you? jsub -l release=trusty -N wpx_project_index -quiet -mem 1G -m a /data/project/projanalysis/bin/python /data/project/projanalysis/wikiproject_scripts/project_index.py [13:50:01] manually installing the right master probably also works [13:50:04] (note the -m a) [13:50:45] hare: yes, should work [13:50:57] andrewbogott: no we don't, it seems weird to have saltstack repos in the jessie base image though [13:51:01] hare: but maybe check if it uses a sane email address with qstat -j after submission [13:51:02] And that will send me an email when the job finishes with a result other than "success" [13:51:17] Tool Labs should have my email address, no? [13:51:40] godog: or even via hiera; salt::master::version = '2014.7.5+ds-1' [13:51:48] https://github.com/wikimedia/operations-puppet/blob/production/modules/salt/manifests/master.pp#L20 [13:52:15] hare: yes, but. [13:52:29] it might use harej@tools-bastion-01.etc, which is unrouteable [13:52:30] valhallasw`cloud: ah, that seems simple enough, I'll give it a try [13:53:15] (is there a way to check this defined email address ahead of time) [13:53:55] hare: not sure [13:54:00] I mean, it should work [13:54:03] if it doesn't, it's a bug [13:54:21] godog: wait, the repo is in the base image? Or just a package? [13:54:33] I'm going to try this with an intentionally broken script >:) [13:55:39] andrewbogott: I couldn't find salt-minion in dpkg.log, that feels like both packages and repo are in the base image [14:02:00] godog: it looks like the image-building tool we use has an automatic salt-install feature. "install_source: stable" [14:02:13] You want me to remove that, or change it to ‘testing’ or something? [14:03:21] it's installing a too-new version (newer than what apt.wm.o has) [14:03:52] it should probably just install whatever apt.wm.o provides? [14:04:05] I fear that removing that will break new instances, as we probably try to set up salt certs before the initial puppet run. [14:04:11] yeah I agree, no additional sources.list.d and install salt-minion [14:04:20] from apt.wm.o that is [14:04:38] valhallasw`cloud: I ran a one-line script that doesn't work, tried submitting it through jsub with the -m a parameters, and I didn't get an email :( [14:04:49] ok, I’ll try that [14:05:16] hare: which jobid? [14:05:33] I don't know -- it ended as soon as it started. [14:05:48] hare: jsub should tell you the job id? [14:05:58] I thought, at least [14:06:37] andrewbogott: thanks! [14:06:39] Ah, I used -quiet [14:06:47] Okay I did it again without -quiet. 254482 [14:07:29] gah, accounting log actually doesn't show anything. Sec. [14:08:14] 6Labs: Salt-master version conflicts on jessie labs instances - https://phabricator.wikimedia.org/T110036#1567371 (10Andrew) 3NEW a:3Andrew [14:08:20] mail_list: valhallasw@tools.wmflabs.org looks sane to me [14:08:47] hare: try jsub -m a /home/valhallasw/test.sh ? (which is just a sleep 10) [14:08:52] then qstat -j [14:08:59] and check mail_list [14:09:01] andrewbogott: that looks like a duplicate of https://phabricator.wikimedia.org/T110032 [14:09:41] 6Labs, 6operations: labs salt master on jessie fails to install salt-master - https://phabricator.wikimedia.org/T110032#1567388 (10Andrew) [14:09:42] 6Labs, 5Patch-For-Review: Salt-master version conflicts on jessie labs instances - https://phabricator.wikimedia.org/T110036#1567387 (10Andrew) [14:10:06] mail_list: tools.projanalysis@tools.wmflabs.org [14:10:35] That should be an alias for a real email address, I hope? [14:11:29] hare: that should forward to all members of that service group [14:11:59] as far as I know I'm the only maintainer, so it should at least send me an email [14:12:07] hare: yes, it should [14:12:24] but I ran a script that was intentionally broken, passing the -m a parameters, and I did not get the requisite email. [14:12:31] 6Labs, 10Tool-Labs: Set up lint checks for labs/toollabs - https://phabricator.wikimedia.org/T65687#1567399 (10hashar) [14:12:42] broken = ? [14:12:47] (a one-liner that tries to do something with a variable not defined, causing a NameError) [14:12:56] it might just not qualify as abort [14:13:00] 6Labs, 10Tool-Labs: Set up lint checks for labs/toollabs - https://phabricator.wikimedia.org/T65687#1567402 (10hashar) Feel free to add back #Continuous-Integration-Config whenever the repo has some basic tests. [14:14:42] there's nothing in the mail log, so probably not. [14:14:56] hare: basically, abort = qdel [14:17:21] So if a Python runtime error doesn't qualify as an abort, what *does* it qualify as? [14:19:26] a finished job, I think [14:21:03] -m e (=end) should always send an email. Although maybe not at abort [14:21:04] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1567434 (10yuvipanda) Yeah, looks like a Client Certificate generator + putting it on NFS might be the easiest way to go. I was trying to not use it because: #... [14:21:07] It's SGE, so who knows? :/ [14:21:19] :( [14:21:38] let me check the user manual! [14:21:53] it's a pdf [14:21:55] almost paper [14:22:54] "a - Send email when the job is rescheduled or aborted For example, by using the qdel command." [14:23:39] so it doesn't really care about your exit code, I guess [14:26:21] 6Labs, 6operations, 5Patch-For-Review: labs salt master on jessie fails to install salt-master - https://phabricator.wikimedia.org/T110032#1567462 (10fgiunchedi) for already existing instances, as suggested by @valhallasw, this can be fixed via project-wide hiera with `"salt::master::salt_version": 2014.7.5+... [14:31:51] so, I can verify that the email address is properly configured [14:32:20] \o/ [14:32:20] Now, to test with my intentionally broken script. [14:32:47] Email is sent. It says the exit status is 2, but not *why*. [14:39:21] hare: that output is in your job.err? [14:39:50] I don't think SGE can mail you your output [14:40:04] it is in my job.err, but the point is that I want to be emailed when something goes wrong [14:40:25] this is all we have... [14:40:47] basically, -m a tells you when SGE breaks [14:40:51] and .err tells you if your code broke [14:42:06] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1567549 (10scfc) You don't need to involve NFS (directly). You can set up the certificate generator on any machine as a HTTP/whatever service, do the auth* via... [14:42:27] so basically I should write my own job manager that emails me when there's an exception? [14:44:33] run_script.sh: your_python_script || mail error.log [14:45:02] but yes, because it's the task of the job manager to run your job, not to guess when it should send an email with content (and with which content) [14:45:12] do you want all of the error log? just the last 100 lines? etc [14:45:44] Presumably I would want the contents of stderr? [14:47:27] "if stderr > 0, send mail" [14:47:34] or the length of stderr [14:47:38] if it is something other than empty [14:51:08] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109, 3Labs-Sprint-111: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1567611 (10yuvipanda) [14:51:16] 6Labs, 3Labs-Sprint-108, 3Labs-Sprint-109, 3Labs-Sprint-111, 5Patch-For-Review: Simple method to have a per-project debian repository - https://phabricator.wikimedia.org/T104194#1567612 (10yuvipanda) [14:53:20] 6Labs: Get instance block-migration working reliably; script and document - https://phabricator.wikimedia.org/T106146#1567624 (10Andrew) [14:53:22] 6Labs, 6operations, 3Labs-Sprint-107, 3Labs-Sprint-108, and 3 others: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1567621 (10Andrew) 5Open>3Resolved a:3Andrew All labvirt hosts are now running 3.16 kernels, and puppet now actively excludes the known-buggy k... [14:53:47] 6Labs, 3Labs-Sprint-105: Upgrade labs network node to trusty - https://phabricator.wikimedia.org/T90823#1567636 (10Andrew) [14:54:46] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-111: Support cold-migration or suspended migration, or something, between labvirt hosts - https://phabricator.wikimedia.org/T109902#1567637 (10Andrew) [14:55:31] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-111: Labs virt capacity expansion - https://phabricator.wikimedia.org/T107624#1567643 (10Andrew) [14:55:51] 6Labs: Upgrade labs cluster to Trusty - https://phabricator.wikimedia.org/T90821#1567645 (10Andrew) 5Open>3Resolved [14:55:51] 6Labs, 10Labs-Infrastructure: Upgrade Labs to Openstack Juno - https://phabricator.wikimedia.org/T104587#1567647 (10Andrew) [14:55:53] 6Labs: Get instance block-migration working reliably; script and document - https://phabricator.wikimedia.org/T106146#1567646 (10Andrew) [14:56:42] 6Labs: Upgrade labs cluster to Trusty - https://phabricator.wikimedia.org/T90821#1068320 (10Andrew) [14:56:43] 6Labs, 3Labs-Sprint-105: Upgrade labs network node to trusty - https://phabricator.wikimedia.org/T90823#1567657 (10Andrew) 5Open>3Resolved labnet1002 now running trusty and handling labs networking. [14:58:51] 6Labs, 10Labs-Infrastructure: Update Labs to OpenStack Kilo - https://phabricator.wikimedia.org/T110045#1567669 (10Andrew) 3NEW a:3Andrew [14:59:23] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-111: Update Labs to OpenStack Juno - https://phabricator.wikimedia.org/T110047#1567686 (10Andrew) 3NEW a:3Andrew [15:10:54] 6Labs, 10Labs-Infrastructure: Switch to a multi_host nova network - https://phabricator.wikimedia.org/T107731#1567749 (10Andrew) p:5Triage>3Lowest This probably won't happen -- we got our upgrade without switching models, and ideally we'll move to Neutron rather than doing this. [15:40:43] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1567858 (10hashar) 3NEW [15:52:08] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1567875 (10hashar) [15:52:30] if anyone around I got two instances in 'paused' state [15:52:41] despite hard rebooting / shutting them down :-/ [15:53:34] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-111: re-image labnet1001 with Trusty - https://phabricator.wikimedia.org/T110053#1567884 (10Andrew) 3NEW [15:57:07] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1567906 (10hashar) @andrew do you have any spare time to look at them please ? :-} [16:12:49] hashar: I will look… do you happen to know if their deaths corresponded with one of the reboots? [16:13:03] !log tools killing all processes of tools.cobain which are flooding tools-bastion-01 [16:13:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [16:15:38] !log tools kill -9'ing because normal killing doesn't work [16:15:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [16:15:43] hashar: that virt host is out of disk space. I’ve been working on that problem for a while but the problem got worse before I found a good fix. Is it easy for you to destroy and rebuild those instances? [16:17:38] andrewbogott: there is a bunch of manual tasks involved [16:17:47] but we that is not the end of the world :-) [16:18:03] can you reply back on the task please? Will recreate them tmorrow [16:19:00] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1567958 (10Andrew) This is probably because they are hosted on labvirt1007 which no longer has space... [16:20:01] Could this be the reason, why I can't access my instance at labvirt1007? [16:25:23] Luke081515: maybe but probably not :) [16:25:56] ok :) [16:30:19] 6Labs: Ignored file /etc/apt/apt.conf.d/20auto-upgrades.ucf-dist - https://phabricator.wikimedia.org/T110055#1568018 (10scfc) 3NEW [16:30:33] YuviPanda, how do I disable a service group? [16:30:53] valhallasw`cloud: uh, don't think you cna... [16:30:55] *can [16:31:02] the instance running Phragile is also down and not booting anymore. is something broken? [16:33:36] andrewbogott: ^ [16:33:39] anything I can help with? [16:33:50] YuviPanda: I don’t think so, I’m working on it as best I can [16:33:58] unless you see things on labvirt1007 you can delete. [16:34:23] I'll look at that [16:34:48] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1568045 (10hashar) At worth I will recreate tomorrow morning Europe time :-) Thanks Andrew! [16:37:26] !log tools more processes were started, so added a talk page message on [[User:Coet]] (who was starting the processes according to /var/log/auth.log) and using 'write coet' on tools-bastion-01 [16:37:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [16:38:07] YuviPanda: that's... problematic [16:38:14] but I think my `write` helped [16:38:55] valhallasw`cloud: we can disable the user who is doing it by getting rid of their shell... [16:39:09] YuviPanda: yes, except this was a service group with 8 or so users [16:39:35] but the auth log at least shows whodunnit [16:42:48] so are labvirt1007 instances not accessible right now? staging-test-tin in the staging project is certainly inaccessable :( [16:45:02] well, I guess I can get to some labvirt1007 instances, just not that one... [16:58:39] problems seems solved: https://wikitech.wikimedia.org/wiki/User_talk:Merlijn_van_Deen#re:tools.cobain_processes_on_tools-bastion-01 [17:02:53] Luke081515: thanks [17:03:27] it's hard to give feedback to users currently doing something :/ but I think email + wikitech + write might be the best we have [17:04:52] !log k8s-eval killed k8s-worker-04 to make space on labvirt1007 [17:05:20] !log mobile kill instance android-build to make space on labvirt1007 (android-builder is the successor) [17:05:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL, Master [17:06:11] YuviPanda: The first message was not logged [17:06:20] yeah [17:06:24] but stashbot will have it [17:06:27] and that's what I use anyway [17:06:46] (http://tools.wmflabs.org/sal/projects) [17:17:22] !log revscoring, delete labels-test, was unused and we needed space on labvirt1007 [17:17:23] revscoring, is not a valid project. [17:31:32] !log staging deleted instance staging-mc3, redundant and should help provide space on labvirt1007 [17:31:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Staging/SAL, Master [17:46:34] !log k8s-eval rebooted k8s-master-01 from nova, it was wedged in Paused state because labvirt1007 ran out of disk space [17:58:26] 6Labs, 6Release-Engineering: Cleanup quaity-assurance labs project - https://phabricator.wikimedia.org/T108087#1568427 (10yuvipanda) Ok, deleting! [18:02:18] !log quality-assurance project deleted, as per https://phabricator.wikimedia.org/T108087 [18:02:18] quality-assurance is not a valid project. [18:02:31] 6Labs, 6Release-Engineering: Cleanup quaity-assurance labs project - https://phabricator.wikimedia.org/T108087#1568458 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Deleted! [19:05:10] how canI kill a process? I tried "kill ", but it still appear when I type "top -u coet" or "ps u | grep "... [19:12:28] coet|cawiki: kill -9? [19:13:34] YuviPanda: I reallized it is a stopped proccess... [19:14:30] ok, it works. YuviPanda [19:14:39] thanks [19:50:15] 6Labs: Ignored file /etc/apt/apt.conf.d/20auto-upgrades.ucf-dist - https://phabricator.wikimedia.org/T110055#1568969 (10hashar) @scfc idea seems quite likely and it is quite trivial to verify: just look whether the image building host has /etc/apt/apt.conf.d/20auto-upgrades.ucf-dist If so delete it on the host... [19:51:24] 6Labs, 10Labs-Infrastructure, 6operations: disk space on labvirt1007 - https://phabricator.wikimedia.org/T109752#1568982 (10hashar) [19:51:27] 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore - https://phabricator.wikimedia.org/T110052#1568981 (10hashar) [20:00:32] gifti: your process '/usr/bin/tclsh8.6 ./wl.tcl' is using quite some memory and cpu time on tools-bastion-01. Would it be possible to run it on the grid instead? [20:03:22] YuviPanda: I built something new! \o/ try /home/valhallasw/pstricks/pstricks.py :-) [20:03:34] valhallasw`cloud: wooo nice! [20:03:38] what is it? :D [20:03:52] heavy process/user report (> 3 min cpu time, > 100MB memory for a process, >250MB memory total for a user) [20:04:28] nice! [20:14:03] (03CR) 10Multichill: [C: 031] "Looks like a good addition." (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233340 (https://phabricator.wikimedia.org/T110003) (owner: 10Jean-Frédéric) [20:16:39] <_joe_> valhallasw`cloud: for OGE? [20:16:51] _joe_: no, just for processes on tools-bastion [20:17:02] <_joe_> oh ok, so local processes [20:17:15] <_joe_> nice anyways :) [20:17:22] I'm also planning to do some stats based on the accounting file at some point [20:17:43] but the main point of this script was to keep tools-bastion-01 a place to log in ;-) [20:28:42] YuviPanda: is tools.admin maintained manually? [20:28:55] as in the service group? [20:28:55] valhallasw`cloud: there's a git pull in an puppet manifest somewhere [20:28:58] valhallasw`cloud: yeah [20:28:59] it is [20:30:49] hmm. okay, except my plan fails because shinken mails me personally [20:30:50] oh well [20:58:39] 6Labs: Support bare-metal server allocation in labs - https://phabricator.wikimedia.org/T95185#1569122 (10GWicke) Another question relevant to services like Revscoring (T106867) is access from production to these semi-production services. Currently, the labs network is not accessible from production, and we migh... [21:22:55] what's the beta equivalent for terbium? deployment-bastion? [21:27:30] tgr: #wikimedia-releng would probably know better [22:07:05] YuviPanda, when is Coren coming back? [22:08:06] Cyberpower678: next week I think [22:08:42] I thought it was this week. [22:08:57] it will be this week. [22:08:58] next week :) [22:09:29] what JohnFLewis said [22:21:51] (03PS3) 10Jean-Frédéric: Support for Wikidata while guessing categories from CommonsCat [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233340 (https://phabricator.wikimedia.org/T110003) [22:28:15] !log cold-migrating ores-web-02 to labvirt1004. [22:28:16] cold-migrating is not a valid project. [22:29:03] (03CR) 10Jean-Frédéric: "Fixed a bit Multichill concern. I think I will merge this :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233340 (https://phabricator.wikimedia.org/T110003) (owner: 10Jean-Frédéric) [22:29:10] !log ores cold-migrating ores-web-02 to labvirt1004. [22:29:23] (03CR) 10Jean-Frédéric: [C: 031 V: 031] "Fixed a bit Multichill concern. I think I will merge this :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/233340 (https://phabricator.wikimedia.org/T110003) (owner: 10Jean-Frédéric)