[06:40:47] <a9309131>	 Damianz: cluebot-redis-relay still down.
[06:44:30] <shinken-wm_>	 RECOVERY - Free space - all mounts on tools-exec-1206 is OK All targets OK
[07:01:43] <wikibugs>	 6Labs, 10Labs-Other-Projects, 6operations: labstore1003 alerting because of network saturation - https://phabricator.wikimedia.org/T110881#1613196 (10Nemo_bis) 5Open>3Resolved a:3Nemo_bis (Note, creating a report with me cc'd doesn't produce notifications, per T107552.)  What's the limit? [[https://gan...
[09:56:40] <wikibugs>	 6Labs, 7Database: Replica database tables may need to be ANALYZEd again - https://phabricator.wikimedia.org/T111656#1613398 (10jcrespo) p:5Triage>3Low
[09:56:47] <wikibugs>	 6Labs, 7Database: Replica database tables may need to be ANALYZEd again - https://phabricator.wikimedia.org/T111656#1613401 (10jcrespo) a:3jcrespo
[10:41:12] <grrrit-wm>	 (03PS1) 10Addshore: Report ArticlePlaceholder changed to #wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/236527 (https://phabricator.wikimedia.org/T109927) 
[10:43:24] <grrrit-wm>	 (03PS1) 10Addshore: Repirt WikimediaBadges changes to #wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/236528 (https://phabricator.wikimedia.org/T109927) 
[10:44:27] <grrrit-wm>	 (03PS2) 10Addshore: Report WikimediaBadges changes to #wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/236528 (https://phabricator.wikimedia.org/T109927) 
[10:44:39] <addshore>	 Repirt!
[10:46:40] <grrrit-wm>	 (03PS1) 10Addshore: Report ArticlePlaceholder to #wikidata-feed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/236530 (https://phabricator.wikimedia.org/T109927) 
[10:48:45] <grrrit-wm>	 (03PS1) 10Addshore: Report Wikidata.org to #wikidata-feed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/236532 (https://phabricator.wikimedia.org/T109927) 
[11:05:55] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] Report ArticlePlaceholder changed to #wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/236527 (https://phabricator.wikimedia.org/T109927) (owner: 10Addshore)
[11:05:58] <grrrit-wm>	 (03Merged) 10jenkins-bot: Report ArticlePlaceholder changed to #wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/236527 (https://phabricator.wikimedia.org/T109927) (owner: 10Addshore)
[11:06:02] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] Report WikimediaBadges changes to #wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/236528 (https://phabricator.wikimedia.org/T109927) (owner: 10Addshore)
[11:06:05] <grrrit-wm>	 (03Merged) 10jenkins-bot: Report WikimediaBadges changes to #wikidata-feed [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/236528 (https://phabricator.wikimedia.org/T109927) (owner: 10Addshore)
[11:06:07] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] Report ArticlePlaceholder to #wikidata-feed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/236530 (https://phabricator.wikimedia.org/T109927) (owner: 10Addshore)
[11:06:13] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] Report Wikidata.org to #wikidata-feed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/236532 (https://phabricator.wikimedia.org/T109927) (owner: 10Addshore)
[11:06:14] <addshore>	 cheers valhallasw`cloud :D
[11:06:18] <addshore>	 ping me once deployed ;)
[11:06:22] <grrrit-wm>	 (03Merged) 10jenkins-bot: Report ArticlePlaceholder to #wikidata-feed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/236530 (https://phabricator.wikimedia.org/T109927) (owner: 10Addshore)
[11:06:24] <addshore>	 pretty please!
[11:06:25] <valhallasw`cloud>	 wikibugs autodeploys ;-)
[11:06:26] <grrrit-wm>	 (03Merged) 10jenkins-bot: Report Wikidata.org to #wikidata-feed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/236532 (https://phabricator.wikimedia.org/T109927) (owner: 10Addshore)
[11:06:32] <addshore>	 oh, nice :D
[11:06:38] <addshore>	 not grrrrt though? :P
[11:06:42] <valhallasw`cloud>	 no :(
[11:06:47] <addshore>	 sad times
[11:07:01] * valhallasw`cloud waits for teh jenkins
[11:07:17] <valhallasw`cloud>	 hm it's merged already
[11:07:25] <addshore>	 speedy
[11:07:31] <ircnotifier>	 !log tools.lolrrit-wm valhallasw: Deployed 4c9f323ff3baf90dd4a52ea12e5178cf5bf66ef4 Report WikimediaBadges changes to #wikidata-feed
[11:07:34] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master
[11:07:44] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 04c06838cc50d916c6cc11b20776a62b8b5fbdc1 Report ArticlePlaceholder to #wikidata-feed
[11:07:46] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master
[11:08:03] <ircnotifier>	 !log tools.lolrrit-wm valhallasw: Deployed 4c9f323ff3baf90dd4a52ea12e5178cf5bf66ef4 Report WikimediaBadges changes to #wikidata-feed
[11:08:05] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master
[11:08:18] * valhallasw`cloud eyes grrrit-wm1
[11:08:19] <valhallasw`cloud>	 GO AWAY.
[11:08:21] <addshore>	 xD
[11:09:05] <addshore>	 thats all of them? :P
[11:09:19] <valhallasw`cloud>	 yeah
[11:09:21] <addshore>	 epic!
[11:10:05] <valhallasw`cloud>	 except gridengine is still a mess it seems
[11:10:21] <valhallasw`cloud>	 WAT.
[11:10:53] <addshore>	 :D
[11:10:57] <addshore>	 so many grrrrts
[11:10:58] <valhallasw`cloud>	 I seriously don't know what SGE is doing
[11:11:06] <valhallasw`cloud>	 the job was just happily running on two hosts
[11:11:10] <valhallasw`cloud>	 only one reported on qstat
[11:18:10] <YuviPanda>	 valhallasw`cloud: I wonder if gridengine is just struggling with the number of hosts we have
[11:18:22] <valhallasw`cloud>	 I dunno. Maybe the killscript is broken?
[11:21:07] <valhallasw`cloud>	 we should just move the ircbots to their own project :P
[11:21:15] <valhallasw`cloud>	 no nfs, no sge, no problems :>
[11:21:50] <YuviPanda>	 we can move them to kubernetes :P
[11:22:00] <YuviPanda>	 not webservices, hmm
[11:22:06] <valhallasw`cloud>	 also not a bad idea
[11:22:13] <valhallasw`cloud>	 yeah, so it's even easier! :P
[11:22:35] <YuviPanda>	 heh
[11:22:35] <YuviPanda>	 yeah
[13:10:38] <valhallasw`cloud>	 addshore: wait, now they are both gone?! :|
[13:10:54] <valhallasw`cloud>	 ftupid
[13:10:59] <addshore>	 :(
[14:03:42] <Niharika>	 Hi, I have a job that's running forever. ps auxf on the host shows it's running, but it's been forty minutes, which is quite unusual. The TIME column in ps aux output shows 0:00 for the job. Is that indicating a problem?
[14:04:47] <valhallasw`cloud>	 Niharika: it might be waiting for user input? check the job .out and .err files
[14:07:32] <Niharika>	 valhallasw`cloud: No, not waiting for input. I've tested the same script on a smaller wiki and it runs in seconds, but doesn't for enwiki. All it does is run a database query and publish results on a wiki page. 
[14:07:46] <Niharika>	 valhallasw`cloud: Issue with memory perhaps?
[14:08:31] <valhallasw`cloud>	 enwiki is massive compared to other wikis, so it could just have a much longer runtime
[14:08:53] <Niharika>	 valhallasw`cloud: Okay, does the TIME column being 0:00 indicate anything? 
[14:09:00] <valhallasw`cloud>	 try running SHOW PROCESSLIST   in   sql enwiki_p
[14:09:28] <valhallasw`cloud>	 I think TIME is user + system time, which would be close to zero if it's waiting for something (such as the query)
[14:11:09] <Niharika>	 valhallasw`cloud: It shows my processes in state "Sending data"
[14:11:22] <Niharika>	 What does that mean?
[14:11:39] <YuviPanda>	 valhallasw`cloud: I wonder if we should allow users to strace their own processes 
[14:11:45] <valhallasw`cloud>	 https://stackoverflow.com/questions/10347193/what-does-it-mean-when-mysql-is-in-the-state-sending-data
[14:11:49] <valhallasw`cloud>	 YuviPanda: I think they can?
[14:12:32] <valhallasw`cloud>	 hm, no, they can't
[14:12:42] <Niharika>	 valhallasw`cloud: Thanks. 
[14:13:39] <valhallasw`cloud>	 YuviPanda: apparently a bad idea: https://www.kernel.org/doc/Documentation/security/Yama.txt ?
[14:13:45] <YuviPanda>	 valhallasw`cloud: yeah kernel switched to it being off by default 
[14:14:45] <YuviPanda>	 Yeah but I wonder if it makes sense for our exec nodes alone 
[14:59:42] <wikibugs>	 6Labs, 10Beta-Cluster: Figure out why wikipedia requires an extra DNS entry that the other sites do not - https://phabricator.wikimedia.org/T111661#1614013 (10hashar) Which entry are you referring to?  Is that the DNS configuration, if so deployment-cache-text04 has: ``` *.beta.wmflabs.org beta-hhvm.wmflabs.or...
[15:06:29] <wikibugs>	 6Labs, 10Beta-Cluster: Figure out why wikipedia requires an extra DNS entry that the other sites do not - https://phabricator.wikimedia.org/T111661#1614035 (10hashar) Found some reference in an IRC log from Feb 2013: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/20130205.txt  ``` lang=irc <andrewbogot...
[15:06:39] <wikibugs>	 6Labs, 10Beta-Cluster: Figure out why wikipedia requires an extra DNS entry that the other sites do not - https://phabricator.wikimedia.org/T111661#1614036 (10hashar) p:5Triage>3Low
[15:12:42] <shinken-wm_>	 PROBLEM - Puppet failure on tools-webproxy-02 is CRITICAL 20.00% of data above the critical threshold [0.0]
[15:13:10] <YuviPanda>	 ^ that is me
[15:27:41] <shinken-wm_>	 RECOVERY - Puppet failure on tools-webproxy-02 is OK Less than 1.00% above the threshold [0.0]
[15:30:57] <apergos>	 on wdq-beta, /var/log/wdqs-updater.log is full of exceptions which has filled the disk, I'll keep a few around but most has to get tossed
[15:37:40] <apergos>	 the same is true on wdq-beta2; I have truncated the logs on both hosts
[16:00:35] <wikibugs>	 6Labs, 10Beta-Cluster: Figure out why wikipedia requires an extra DNS entry that the other sites do not - https://phabricator.wikimedia.org/T111661#1614160 (10Krenair) a:3Andrew @andrew: Please set up the missing $project-beta => $project.beta.wmflabs.org entries. This requires cloudadmin to do properly thou...
[16:12:36] <shinken-wm_>	 PROBLEM - Puppet failure on tools-services-01 is CRITICAL 22.22% of data above the critical threshold [0.0]
[16:18:06] <YuviPanda>	 ^ that is me
[16:22:35] <valhallasw`cloud>	 Aptly :o
[16:22:55] <valhallasw`cloud>	 YuviPanda: now we just nees build hosts :p
[16:23:08] <YuviPanda>	 valhallasw`cloud: heh.
[16:23:18] <YuviPanda>	 valhallasw`cloud: I'm setting this up because i"m starting to bootstrap a k8s cluster in tools
[16:23:22] <YuviPanda>	 valhallasw`cloud: without NFS
[16:23:46] <YuviPanda>	 valhallasw`cloud: I"m thinking we should offer people to opt-in to run long running processes to begin with, rather than web services. those can come next, I guess
[16:23:52] <JohnFLewis>	 without NFS? :o
[16:24:08] <YuviPanda>	 valhallasw`cloud: or rather, allow people to run arbitrary docker images first, build compat layer on top after
[16:24:36] <YuviPanda>	 JohnFLewis: yes, but just to start with :) 
[16:25:22] <JohnFLewis>	 With the way people use it, it seems impossible to live without NFS! (Sarcastic of course :))
[16:25:32] <YuviPanda>	 heh
[16:25:39] <YuviPanda>	 well we need to offer a lot of alternatives
[16:25:47] <YuviPanda>	 logstash for logging, larger file systems
[16:25:56] <YuviPanda>	 all as transparent as possible, of course
[16:26:00] <gifti>	 yay, testing
[16:26:17] <JohnFLewis>	 YuviPanda: bare metal allocation too :P
[16:26:26] <JohnFLewis>	 Which is great to see on the workflow and in progress
[16:26:59] <YuviPanda>	 JohnFLewis: eventually yeah.
[16:27:39] <shinken-wm_>	 RECOVERY - Puppet failure on tools-services-01 is OK Less than 1.00% above the threshold [0.0]
[16:32:29] <wikibugs>	 6Labs, 10Tool-Labs: Move toollabs off labsdebrepo into aptly - https://phabricator.wikimedia.org/T111708#1614251 (10yuvipanda) 3NEW
[17:38:44] <wikibugs>	 6Labs, 6operations, 5Patch-For-Review: labs salt master on jessie fails to install salt-master - https://phabricator.wikimedia.org/T110032#1614354 (10jcrespo) @Andrew did that last related patch fix the issue or does it require more work?
[17:39:10] <wikibugs>	 6Labs, 6operations, 5Patch-For-Review: labs salt master on jessie fails to install salt-master - https://phabricator.wikimedia.org/T110032#1614355 (10jcrespo) p:5Triage>3Normal
[17:56:08] <Luke081515>	 is the topic if this channel still up to date?
[17:57:12] <YuviPanda>	 Luke081515: updated
[17:57:52] <YuviPanda>	 !log tools created tools-k8s-master-01 with jessie, will be etcd and kubernetes master
[17:57:55] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[17:58:47] <Luke081515>	 "lNew instances"? Or is this a typo?
[17:59:09] <YuviPanda>	 Luke081515: yes :) 
[17:59:15] <Luke081515>	 :)
[18:13:06] <andrewbogott>	 YuviPanda: are you looking at puppet failures already?
[18:13:29] <YuviPanda>	 andrewbogott: I caused them most, all recovered now
[18:13:34] <andrewbogott>	 oh, great :)
[18:13:35] <YuviPanda>	 andrewbogott: enjoy your weekend!
[18:13:47] <andrewbogott>	 btw, do new instances still need a push?
[18:14:11] <YuviPanda>	 andrewbogott: no, I just created one and it seems ok
[18:14:13] <YuviPanda>	 andrewbogott: I'll take it out
[18:14:18] <andrewbogott>	 cool
[18:14:45] <andrewbogott>	 hm, manual puppet runs still failing for me
[18:14:54] <YuviPanda>	 andrewbogott: oh
[18:15:06] <YuviPanda>	 andrewbogott: I see alerts on email now, I was just seeting the ones on IRC
[18:15:10] * YuviPanda waves hands at shinken-wm_
[18:15:19] <YuviPanda>	 andrewbogott: which host are you looking at?
[18:15:40] <andrewbogott>	 tools-webgrid-lighttpd-1401 and tools-exec-1210
[18:15:47] <andrewbogott>	 aaaaaand now it’s better
[18:15:56] <andrewbogott>	 possibly because I restarted things or possibly because it was better already
[18:16:27] <YuviPanda>	 heh
[18:16:43] <YuviPanda>	 andrewbogott: now go back to vacationing!
[18:16:53] <andrewbogott>	 hmmmmm
[18:16:54] <andrewbogott>	 ok :)
[18:17:39] <valhallasw`cloud>	 YuviPanda: ok, so ::package_builder also adds build things for a lot of distros we will never build for. Is that something we care about? I'm not sure how much space they take
[18:18:18] <YuviPanda>	 valhallasw`cloud: how many other distros?
[18:18:44] <valhallasw`cloud>	 precise, trusty, jessie, sid, precise386, trusty386, jessie386 sid386
[18:19:27] <shinken-wm_>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1401 is CRITICAL 44.44% of data above the critical threshold [0.0]
[18:19:51] <shinken-wm_>	 PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL 20.00% of data above the critical threshold [0.0]
[18:22:02] <YuviPanda>	 um
[18:22:04] <YuviPanda>	 I wonder if that's me
[18:22:08] <YuviPanda>	 valhallasw`cloud: should be ok, I guess
[18:23:47] <valhallasw`cloud>	 aaah reverse dependency statements
[18:23:51] <valhallasw`cloud>	 puppet :((
[18:27:55] <valhallasw`cloud>	 YuviPanda: well, let me try applying package_builder to a toolsbeta host
[18:28:01] <YuviPanda>	 valhallasw`cloud: ok!
[18:28:25] <valhallasw`cloud>	 YuviPanda: from the looks of it there are some dependencies wrong, but we'll see
[18:29:28] <shinken-wm_>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1401 is OK Less than 1.00% above the threshold [0.0]
[18:29:50] <shinken-wm_>	 RECOVERY - Puppet failure on tools-exec-1210 is OK Less than 1.00% above the threshold [0.0]
[18:30:21] <valhallasw`cloud>	 !log toolsbeta applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages)
[18:30:24] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL, Master
[18:31:06] <valhallasw`cloud>	 I hope bastion and compute don't bite too mcuh :p
[18:32:36] <Pathoschild>	 Hi. Any idea why the static server often returns HTTP 403 Forbidden for files recently updated from Git? The 403 seems to be cached on the server; after renaming the file, the new name works fine but the old name keeps returning 403 (even though it's not there anymore). This seems to last for quite a while (5+ hours since I started keeping track today).
[18:32:52] <Pathoschild>	 It happens even if I break the symlink from ~/www/static to the git repository before updating, too.
[18:33:13] <Pathoschild>	 Today's example: https://tools-static.wmflabs.org/meta/scripts/edge/pathoschild.regexeditor.css
[18:33:48] <wikibugs>	 6Labs, 6operations, 10wikitech.wikimedia.org, 7HHVM: Move wikitech (silver) to HHVM - https://phabricator.wikimedia.org/T98813#1614471 (10jcrespo) 5Open>3stalled As there is some disagreement here, trying to be neutral here and just reflecting the current state of this task.
[18:34:25] <valhallasw`cloud>	 Pathoschild: eh, there's no /static in your public_html?
[18:34:37] <valhallasw`cloud>	 iirc files had to be in a specific subdir
[18:34:49] <valhallasw`cloud>	 otherwise people would be able to read all your php files etc
[18:34:59] <Pathoschild>	 Yep, under ~/www/static.
[18:35:12] <valhallasw`cloud>	 oh, right
[18:37:28] <valhallasw`cloud>	 hrm.
[18:39:30] <valhallasw`cloud>	 ok, it's not an nfs issue, as the file is not there
[18:40:46] <valhallasw`cloud>	 there's also no 'expires' setting
[18:41:24] <valhallasw`cloud>	 Pathoschild: restarted nginx, still the same issue. Very weird...
[18:41:55] <Pathoschild>	 Yep. It works fine with the new name: https://tools-static.wmflabs.org/meta/scripts/edge/pathoschild.regexeditor2.css
[18:42:00] <valhallasw`cloud>	 ...wrong static host. Ok, let's try that agian.
[18:42:34] <valhallasw`cloud>	 nope, still a 403 :/
[18:42:43] <valhallasw`cloud>	 and a non-existent file gives a 4040
[18:42:44] <valhallasw`cloud>	 404*
[18:42:56] <Pathoschild>	 At first I thought Git might temporarily change the permissions, and that gets cached somewhere. But it happens even if I unsymlink the repository, update the files, and only resymlink when it's done.
[18:44:00] <valhallasw`cloud>	 Pathoschild: actually, it does seem to be nfs
[18:44:14] <valhallasw`cloud>	 valhallasw@tools-web-static-01:/data/project/meta/www/static/scripts/edge$ cat pathoschild.regexeditor.css
[18:44:14] <valhallasw`cloud>	 cat: pathoschild.regexeditor.css: Permission denied
[18:44:14] <valhallasw`cloud>	 valhallasw@tools-web-static-01:/data/project/meta/www/static/scripts/edge$ cat pathoschild.regexeditor.csss
[18:44:14] <valhallasw`cloud>	 cat: pathoschild.regexeditor.csss: No such file or directory
[18:44:17] <valhallasw`cloud>	 very weird
[18:45:38] <valhallasw`cloud>	 !log tools weird NFS issue on tools-web-static-01. Switching over to -02 before rebooting.
[18:45:42] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[18:45:52] <Pathoschild>	 Weird. I get the standard "no such file or directory" when I do it from tools-bastion-01.
[18:46:05] <valhallasw`cloud>	 yeah, which is why I blame NFS
[18:46:17] <valhallasw`cloud>	 I can try a remount, actually
[18:46:29] <valhallasw`cloud>	 still, let me switch to -02
[18:47:28] <valhallasw`cloud>	 ok, https://tools-static.wmflabs.org/meta/scripts/edge/pathoschild.regexeditor.css is 404 now
[18:47:32] <valhallasw`cloud>	 (other static webserver)
[18:47:45] <valhallasw`cloud>	 !log tools switched static webserver to tools-static-02
[18:47:48] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[18:49:45] <valhallasw`cloud>	 !log tools ran sudo mount -o remount /data/project    on tools-static-01, which also solved the issue, so skipping the reboot
[18:49:48] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[18:49:49] <wikibugs>	 6Labs, 10Tool-Labs: Make tools-instances that don't need NFS not have NFS - https://phabricator.wikimedia.org/T111716#1614517 (10yuvipanda) 3NEW
[18:50:53] <valhallasw`cloud>	 !log toolsbeta role::bastion is now applied on -exec-101. Now for the package_builder manifest...
[18:50:56] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL, Master
[18:53:51] <Pathoschild>	 Thanks! I'll let you know whether it keeps happening next time I update the repo.
[18:54:20] <valhallasw`cloud>	 thanks!
[19:18:32] <wikibugs>	 6Labs, 7Database: Replica database tables may need to be ANALYZEd again - https://phabricator.wikimedia.org/T111656#1614559 (10jcrespo) 5Open>3Resolved Resolved, that query on labsdb1002 and labsdb1003 now runs in 7-9 seconds.  Do not doubt on reporting any other similar issue. Although if it happens very...
[19:24:26] <wikibugs>	 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1614590 (10jcrespo)
[19:24:28] <wikibugs>	 6Labs, 10Tool-Labs, 7Database: Potential templatelinks data integrity issue on Tool Labs' enwiki_p - https://phabricator.wikimedia.org/T111371#1614585 (10jcrespo) 5Open>3Resolved I am building an automatic checker on T104459. Sadly I can only work on that when there is nothing else more important. This s...
[19:31:17] <valhallasw`cloud>	 YuviPanda: aaaaaah there's also ensure_packages which is sort or require_package but in stdlib and not quite as good
[19:31:20] * valhallasw`cloud sobs
[19:32:31] <YuviPanda>	 valhallasw`cloud: yes
[19:32:32] <YuviPanda>	 :(
[19:32:33] <YuviPanda>	 i know
[19:35:01] <wikibugs>	 6Labs, 7Database: Measure capacity and utilization of labsdb*** boxes - https://phabricator.wikimedia.org/T107070#1614611 (10jcrespo) p:5Triage>3Low Low right now, I haven't even started evaluating this even on production, and labsdbs have been working acceptably well in the last months.
[19:38:09] <Krinkle>	 YuviPanda: Yes or no, bigbrotherrc is still working and the recommended way to ensure jstart
[19:38:26] <YuviPanda>	 unfortunately yes.
[19:38:36] <YuviPanda>	 only for jobs, however
[19:38:38] <Krinkle>	 In that case, it ain't working
[19:38:40] <YuviPanda>	 webservices should 'just work'
[19:38:42] <Krinkle>	 tools.wmfdbbot
[19:38:57] <Krinkle>	 Almost every week I learn the bot has been down for several days
[19:39:04] <Krinkle>	 and need to start it manually
[19:39:21] <Krinkle>	 It claims to be running?
[19:39:22] <Krinkle>	   37336 0.42528 dbbot-wm   tools.wmfdbb r     08/18/2015 14:57:03 continuous@tools-exec-1202.eqi     1         
[19:39:26] <YuviPanda>	 can you file a bug? I've to go now, unforunately (last night in berlin)
[19:39:31] <Krinkle>	 k
[19:39:38] <YuviPanda>	 Krinkle: yeah, gridengine itself has been wonky and seeing thse ghost jobs
[19:39:45] <Krinkle>	 I'm going to SF in 2.5 week
[19:39:47] <YuviPanda>	 which claim to be running, so bigbrother and ?status page think it's ok
[19:39:52] <Krinkle>	 You better be back before then :P
[19:39:57] <YuviPanda>	 Krinkle: I'm back on wednesday :D
[19:40:02] <Krinkle>	 kk :)
[19:40:09] <YuviPanda>	 and I'm going to be there for 6 months!
[19:51:29] <valhallasw`cloud>	 YuviPanda: is the toollabs-webservice deb new? it's not on toollabs
[19:51:59] <valhallasw`cloud>	 or maybe nfs is still broken
[19:52:00] <valhallasw`cloud>	 *checks*
[19:54:53] <valhallasw`cloud>	 YuviPanda: ok, made dpkg tool to make that easier as well :-p http://tools-static.wmflabs.org/dpkg/deb-precise/
[20:02:19] <valhallasw`cloud>	 YuviPanda: AAAAAAAAAAARGH
[20:02:20] <valhallasw`cloud>	 so
[20:02:27] <valhallasw`cloud>	 require_package makes the class require those packages
[20:02:38] <valhallasw`cloud>	 but all our stuff inherits class toollabs
[20:02:44] <valhallasw`cloud>	 and that's where the labsdebrepo is configured
[20:02:47] <valhallasw`cloud>	 see the issue here? :-p
[20:03:12] <YuviPanda>	 valhallasw`cloud: hahahahaha :'(
[20:03:24] <YuviPanda>	 valhallasw`cloud: so clearly step 1 is to move to aptly :D
[20:03:30] <valhallasw`cloud>	 well, no, that's the same problem
[20:03:35] <YuviPanda>	 all toollabs hosts already have aptly configured
[20:03:35] <valhallasw`cloud>	 I'm going to make toollabs::apt
[20:03:43] <valhallasw`cloud>	 and make class toollabs depend on thta
[20:03:44] <YuviPanda>	 and you won't have a require cycle
[20:03:45] <valhallasw`cloud>	 and that should fix it
[20:03:51] <valhallasw`cloud>	 eh
[20:03:57] <valhallasw`cloud>	 how do they have aptly configured?
[20:04:20] <YuviPanda>	 valhallasw`cloud: http://wikitech.wikimedia.org/wiki/Hiera:tools
[20:04:21] <YuviPanda>	 classes
[20:04:37] <valhallasw`cloud>	 oh, right.
[20:04:45] <valhallasw`cloud>	 other set of modules so not an issue
[20:05:00] <YuviPanda>	 ya
[20:05:25] <valhallasw`cloud>	 except my extra class won't solve the order of dependencies. argh.
[20:09:16] <valhallasw`cloud>	 bah, bah, bah again
[20:09:40] * YuviPanda hugs valhallasw`cloud, offers stroopwafel
[20:09:59] <valhallasw`cloud>	 YuviPanda: I'm going to push the WIP and not look at it for a month
[20:10:05] <YuviPanda>	 heh
[20:10:16] <YuviPanda>	 I can try moving us to aptly tomorrow
[20:10:40] <valhallasw`cloud>	 and aptly has   aptly_repo -> Package <| |>   set? :-p
[20:10:51] <YuviPanda>	 no but doesn't need to?
[20:11:02] <valhallasw`cloud>	 of course it does
[20:11:07] <YuviPanda>	 why does it?
[20:11:09] <valhallasw`cloud>	 because otherwise it won't find packages sometimes
[20:11:21] <YuviPanda>	 no, puppet-run has an apt-get update prepended to it
[20:11:27] <YuviPanda>	 so after first run it should all just work
[20:11:43] <valhallasw`cloud>	 meh, maybe.
[20:18:01] <valhallasw`cloud>	 YuviPanda: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Package[build-essential] is already declared; cannot redeclare at /etc/puppet/modules/package_builder/manifests/init.pp:20 on node toolsbeta-exec-101.toolsbeta.eqiad.wmflabs
[20:18:10] <valhallasw`cloud>	 (╯°□°）╯︵ ┻━┻)
[20:20:20] <valhallasw`cloud>	 I don't even know how to debug that
[20:20:25] <valhallasw`cloud>	 they are both require_package()s
[20:20:32] <valhallasw`cloud>	 oh no, they are not
[20:20:38] <valhallasw`cloud>	 package_builder uses ensure_packages
[20:31:01] <wikibugs>	 6Labs, 10Tool-Labs: deploy package_builder on tool labs - https://phabricator.wikimedia.org/T111730#1614813 (10valhallasw) 3NEW
[20:35:43] <ankita-ks>	 YuviPanda : around?
[20:35:56] <ankita-ks>	 I have a mediawiki-vagrant setup on labs
[20:35:57] <YuviPanda>	 Am out and on my phone...
[20:36:02] <YuviPanda>	 So... Kinda? :)
[20:36:03] <YuviPanda>	 Ah
[20:36:07] <wikibugs>	 6Labs, 10Tool-Labs: deploy package_builder on tool labs - https://phabricator.wikimedia.org/T111730#1614827 (10valhallasw) Last point should be fixable by using a Trusty host.
[20:36:11] <ankita-ks>	 I enabled the role visualeditor
[20:36:13] <ankita-ks>	 ah okay
[20:36:24] <ankita-ks>	 But when I try to get into edit mode
[20:36:58] <ankita-ks>	 I get the error : Error loading data from the server: 404: parsoidserver-http: HTTP 404. Would you like to try again?
[20:37:35] <ankita-ks>	 Also, when I ran a vagrant provision after enabling visual editor role, I got this :
[20:38:22] <ankita-ks>	 Warning: Scope(Class[Mediawiki::Parsoid]): Could not look up qualified variable 'role::mediawiki::hostname'; class role::mediawiki has not been evaluated
[20:38:50] <ankita-ks>	 bd808 : could you help?
[20:39:12] <YuviPanda>	 Have you asked the VE / parsoid team? bd808 is also the de facto vagrant person. And #wikimedia-dev might also provide more help...
[20:39:28] <ankita-ks>	 I haven't asked anywhere else yet.
[20:39:34] <ankita-ks>	 I will try doing that as well
[20:39:49] <YuviPanda>	 Tried updating the vagrant git repo?
[20:39:57] <YuviPanda>	 It should be in /srv somewhere
[20:40:53] <ankita-ks>	 No, I just set it up by enabling the role via configure option in wikitech 
[20:41:00] <ankita-ks>	 Assumed it's updated
[20:41:06] <ankita-ks>	 Will try doing that now
[20:41:10] <YuviPanda>	 Ok
[20:48:58] <wikibugs>	 6Labs, 10Beta-Cluster, 10Labs-Infrastructure, 7Graphite, 7Shinken: Delete more specific deployment-prep graphite datapoints - https://phabricator.wikimedia.org/T111540#1614842 (10hashar)
[21:03:20] <valhallasw`cloud>	 YuviPanda: ok, lotsopatches on gerrit
[21:03:25] <valhallasw`cloud>	 took too much time :(
[21:07:00] <halfak>	 !log wikilabels deployed wikilabels:f810194 and wikilabels-wikimedia-config:d5f59ce
[21:07:03] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master
[21:46:12] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Catch pywikibot.exceptions.InvalidTitle [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/236684 (https://phabricator.wikimedia.org/T111732) 
[21:46:55] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032] Catch pywikibot.exceptions.InvalidTitle [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/236684 (https://phabricator.wikimedia.org/T111732) (owner: 10Jean-Frédéric)
[21:47:01] <grrrit-wm>	 (03Merged) 10jenkins-bot: Catch pywikibot.exceptions.InvalidTitle [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/236684 (https://phabricator.wikimedia.org/T111732) (owner: 10Jean-Frédéric)
[21:48:50] <github>	 [13intuition] 15Krinkle merged 06composer into 06master: 02https://github.com/Krinkle/intuition/compare/b06836c479af...5802ca52568b
[21:52:18] <grrrit-wm>	 (03PS1) 10Jean-Frédéric: Do not use .title() on a page that does not exist [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/236685 (https://phabricator.wikimedia.org/T111732) 
[21:52:33] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032] Do not use .title() on a page that does not exist [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/236685 (https://phabricator.wikimedia.org/T111732) (owner: 10Jean-Frédéric)
[21:52:38] <grrrit-wm>	 (03Merged) 10jenkins-bot: Do not use .title() on a page that does not exist [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/236685 (https://phabricator.wikimedia.org/T111732) (owner: 10Jean-Frédéric)
[22:17:44] <Pathoschild>	 valhallasw`cloud: the NFS permission issue seems to be fixed; I pushed a few updates with no problem so far. Thanks!
[23:03:22] <github>	 [13intuition] 15Krinkle 04force-pushed 06master from 14fd94a5b to 14dd06656: 02https://github.com/Krinkle/intuition/commits/master
[23:03:23] <github>	 13intuition/06master 14dd06656 15Timo Tijhof: Convert framework to a reusable Composer package...
[23:27:41] <wikibugs>	 6Labs, 10Tool-Labs: Install composer on tools-login - https://phabricator.wikimedia.org/T104789#1615175 (10Krinkle) Work-around: ```lang=sh $ ssh tools-login $ become mytool # https://getcomposer.org/download/ $ mkdir bin $ curl -sS https://getcomposer.org/installer | php -- --install-dir=bin --filename=compos...