[00:04:29] <wikibugs>	 3Wikimedia-Labs-wikitech-interface: Many "(Cannot contact the database server: Too many connections (208.80.154.18))" errors - https://phabricator.wikimedia.org/T78492#847171 (10Aklapper) p:5Triage>3High
[01:31:29] <twentyafterfour>	 Is there any way to increase the size of a volume on a labs instance?  I've got only 2gb for /var and logs are eating that up even with logrotate ...
[01:32:14] <Betacommand>	 twentyafterfour: why are your logs eating so much space?
[01:33:06] <twentyafterfour>	 Betacommand: someone's got atop raw binary logging, those files are pretty big ...
[01:33:08] <chasemp>	 twentyafterfour: there is a large var role I think you can switch on but it only affects var log I think
[01:33:28] <chasemp>	 if you know how to add a role it's called large logs or something
[01:33:38] <chasemp>	 I had to poke at this awhile ago for similar reasons
[01:33:41] <twentyafterfour>	 chasemp: and that automatically resizes the volume?
[01:33:45] <Betacommand>	 twentyafterfour: why not just turn that off, or change how it works
[01:33:50] <chasemp>	 that is a question to which I don't know that answer
[01:34:05] <twentyafterfour>	 Betacommand: it's not my thing to shut off
[01:34:26] <twentyafterfour>	 2gb of /var is pretty tight, really. increasing the size is the right solution IMO
[01:35:05] <twentyafterfour>	 other things on /var take a bit of space, so logs is left with just less than 1.5gb to work with
[01:36:26] <twentyafterfour>	 wikitech.wikimedia.olrg is down?
[01:36:32] <twentyafterfour>	 (Cannot contact the database server: Too many connections (208.80.154.18))
[01:37:39] <chasemp>	 seems yes
[01:37:40] <twentyafterfour>	 duh.. already reported T78492
[01:38:19] <twentyafterfour>	 does that use it's own mysql server or is it part of the main mysql cluster?
[01:38:50] <chasemp>	 I think it's a shared db server but not part of main mysql cluster either :)
[01:38:58] <chasemp>	 honestly I've not touched it
[01:39:03] <chasemp>	 cc'd springle there
[01:39:37] <twentyafterfour>	 I'd poke at it but I don't think I have enough access/knowledge to really do much good
[01:40:50] <chasemp>	 in leiu of a "suggested tickets" thing I wouldn't mind doing a maniphest field that is like "Known issues and outages"
[01:41:03] <chasemp>	 that links to kind of big-ish on going things like when oauth is out or wikitech i down
[01:41:14] <chasemp>	 make it easy to find ongoing outages that don't need dupe reports
[01:41:40] <wikibugs>	 3Wikimedia-Labs-wikitech-interface: Many "(Cannot contact the database server: Too many connections (208.80.154.18))" errors - https://phabricator.wikimedia.org/T78492#847207 (10Andrew) Looks like we're getting crawled.  208.80.154.18 - - [15/Dec/2014:01:40:57 +0000] "GET /w/api.php?action=query&prop=revisions&f...
[01:46:51] <andrewbogott>	 twentyafterfour: Regarding /var, it kind of depends on how old your instance is.  Very recent instances use lvm so you can resize the partition.
[01:47:16] <andrewbogott>	 Older instances are messier -- you can create a new partition for /var/log but that will break lots of things since they're all expecting particular subdirs to already exist.
[01:47:34] <andrewbogott>	 So, yeah, reducing your log usage is probably the easiest :(
[01:48:02] <andrewbogott>	 chasemp: the outage is because of a DOS by a web crawler.  I don't immediately know how to block that but if the problem persists I'll dig in
[01:48:37] <chasemp>	 andrewbogott: thanks for looking man!  
[01:49:01] <andrewbogott>	 chasemp: do you know offhand how to troubleshoot something like that?  I used to maintain the robots.txt myself but now that we use the upstream wiki config...
[01:49:47] <chasemp>	 well at my last place we would ngrep for the offending user agent
[01:49:57] <chasemp>	 and then you could block by user agent in the load balancers
[01:50:06] <chasemp>	 here I don't know where the place to do that would be
[01:50:24] <chasemp>	 I don't even know where wikitech is running :)
[01:50:25] <wikibugs>	 3Wikimedia-Labs-wikitech-interface: Many "(Cannot contact the database server: Too many connections (208.80.154.18))" errors - https://phabricator.wikimedia.org/T78492#847208 (10Andrew) Most likely the issue here is that the main cluster robots.txt is more tolerant of crawlers than wikitech can afford to be.  I'...
[01:50:41] <andrewbogott>	 chasemp: virt1000 is a stand-alone server, no balancers involved.
[01:50:50] <andrewbogott>	 oh, wikitech = virt1000
[01:54:28] <chasemp>	 a few things I know to do are, figure out where it's sucking up time and resources and crawling and disallow the path, or setting a crawl delay to throttle it, but all depend on well mannered bots
[01:54:32] <chasemp>	 which this may very well be
[01:56:14] <Betacommand>	 andrewbogott: https://gerrit.wikimedia.org/r/165212 is a patch for labs
[01:56:32] <Betacommand>	 chasemp: these are not well mannered bots
[01:57:34] <andrewbogott>	 hm, 'You do not have permission to view this object.' that shouldn't happen
[01:57:41] <andrewbogott>	 Betacommand: I think we're talking about two different things...
[01:58:09] <Betacommand>	 andrewbogott: I thought it might be relevant 
[02:00:46] <Betacommand>	 andrewbogott: let me see if I can dig something else useful up
[02:01:00] <andrewbogott>	 At least one of the crawlers is from yahoo, 68.180.228.125
[02:02:19] <Krenair>	 andrewbogott, you know about MediaWiki:Robots.txt ?
[02:02:42] <andrewbogott>	 Krenair: I do.  Wikitech is a weird case where it uses the standard mw robots but is running on its own tiny, feeble server.
[02:02:51] <Krenair>	 ah
[02:03:06] <andrewbogott>	 It may be possible to have server-specific config, but I'm not sure.
[02:03:10] <Krenair>	 umm
[02:03:24] <Krenair>	 don't you just want to append more entries to wikitext's MediaWiki:Robots.txt then?
[02:03:39] <andrewbogott>	 Looks like the storm has passed for the moment.  I will try to bug Bryan or Timo about this in the morning.
[02:03:50] <chasemp>	 there is like a mod_qos for cutting the knees out from under any crawler as a hard nosed crawler delay but it's not a trivial thing I want to get into on a late sunday night :)
[02:03:51] <andrewbogott>	 Krenair: yes, but that file is /managed/ by the upstream mw config.
[02:03:54] <Krenair>	 wikitech's*
[02:04:20] <andrewbogott>	 So, the problem exists in the realm of deployment/customization, not so much in 'what to do'
[02:05:15] <chasemp>	 andrewbogott: sorry man, usually it's just throttle or block them :)
[02:05:27] <chasemp>	 no good answer if it's coupled with the mw config atm
[07:09:35] <matanya>	 andrewbogott: morning (or what ever it is by you)
[07:09:50] <andrewbogott>	 3pm -- almost morning :)
[07:10:02] <matanya>	 the daily poke about debian image and more horse power :)
[07:11:14] <matanya>	 andrewbogott: you said that if you need help with the hp's you will ask, since you didn't, i guess you don't need
[07:11:39] <andrewbogott>	 matanya: yeah, I'm seldom stumped but it just takes hours between tests.
[07:11:50] <andrewbogott>	 And I think you already declined to work on partman, right?  :)
[07:12:01] <matanya>	 I haven't, but now i do :)
[07:12:33] <andrewbogott>	 I have a few patches pending for https://github.com/andsens/bootstrap-vz.  I haven't addressed lvm yet, though, which is supported in our latest trusty image.
[07:12:48] <andrewbogott>	 If you're interested in contributing to that project, that's probably the next big thing that we'll want.
[07:13:17] <andrewbogott>	 (Which, I should add:  I barely know how lvm works with ubuntu now, I just know that Coren wrote some crazy hoop-jumping code to make resizeable partitions.)
[07:14:34] <matanya>	 probably a dumb question: why not use btrfs instead, and get rid of the need for lvm ?
[07:15:13] <andrewbogott>	 Does kvm support btrfs?
[07:15:55] <matanya>	 it does, iirc
[07:17:04] <andrewbogott>	 At the moment, bootstrap-vz only supports ext2, ext3, ext4, xfs.  I'd be happy to use btrfs if that turns out to be easier to add than lvm.
[07:17:43] <matanya>	 andrewbogott: from 2012 but: http://serverfault.com/questions/390810/linux-on-linux-kvm-which-filesystem-should-i-use-for-guest-and-host
[07:17:53] <andrewbogott>	 What I did last week was create bugs for that project like "would you like me to add feature X" and Anders was pretty responsive.
[07:18:26] <matanya>	 Don't use the linux filesystem btrfs on the host for the image files. It will result in low IO performance. The kvm guest may even freeze when high IO traffic is done on the guest. 
[07:19:31] <andrewbogott>	 Yeah, but we weren't talking about the host, were we?
[07:21:22] <matanya>	 no, just claryifing that part
[07:22:00] <matanya>	 i guess you can assign storage pools and grow them as needed
[07:22:18] <andrewbogott>	 There's nothing in that thread that strictly rules out using btrfs for guests...
[07:22:32] <matanya>	 right
[07:22:46] <andrewbogott>	 The current system is that we allocate a certain amount of space for an instance, but then partition only a smaller section of that.  Ideally we'd be able to expand those partitions after the fact, as needed.
[07:23:10] <matanya>	 so btrfs sounds ideal in this use case
[07:23:37] <andrewbogott>	 Oh, another (simpler) thing that's missing from bootstrap-vz is the option to make a separate partition for /var or /var/log.  It has three hard-coded partition mappings; there's no reason why it wouldn't take arbitrary ones in addition.
[07:23:56] <andrewbogott>	 btrfs partitions are resizable on the fly, I take it?
[07:24:18] <andrewbogott>	 https://andsens.github.io/bootstrap-vz/manifest.html, section 'Volume'
[07:24:36] <matanya>	 yes, it is
[07:25:32] <matanya>	 andrewbogott: what did you want your backing to be? raw ?
[07:26:59] <andrewbogott>	 We've been using '"disk_format="qcow2" container_format="ovf"'
[07:27:06] <andrewbogott>	 Um… I'm not sure if that answers your question
[07:27:47] <matanya>	  backing: Specifies the volume backing. This setting is very provider specific. ebs,s3,vmdk,vdi,raw 
[07:27:59] <matanya>	 it is required
[07:28:24] <matanya>	 ebs and s3 are for amazon, vmdk and vdi are virtualbox and vmware
[07:28:32] <matanya>	 so you ar left with raw
[07:28:52] <andrewbogott>	 oh yes, raw
[07:28:57] <andrewbogott>	 Here's the manifest I'm using:  https://gerrit.wikimedia.org/r/#/c/179765/2/modules/labs_bootstrapvz/files/labs-jessie.manifest.yaml
[07:28:59] <andrewbogott>	 (work in progress)
[07:30:03] <matanya>	 do yourself a favor and use gpt and not msdos
[07:30:18] <andrewbogott>	 the install_sudo bit is not working yet.
[07:30:34] <andrewbogott>	 ok -- what's the difference?  (I've been using msdos because that's what all the example manifests use)
[07:31:12] <matanya>	 first of all, support for larger than 2 TB drives
[07:31:48] <andrewbogott>	 well, /that's/ not likely to come up :)
[07:32:42] <matanya>	 https://en.wikipedia.org/wiki/GUID_Partition_Table
[07:33:40] <andrewbogott>	 The latest:  https://gerrit.wikimedia.org/r/#/c/179765/3/modules/labs_bootstrapvz/files/labs-jessie.manifest.yaml
[07:35:20] <matanya>	 andrewbogott: are you using grun <2 ?
[07:35:23] <matanya>	 *grub
[07:36:00] <andrewbogott>	 matanya: I don't think so, but on the other hand those lines to edit the menu don't work.
[07:36:05] <andrewbogott>	 So, I'd welcome a fix for that
[07:36:09] <andrewbogott>	 Speeds up boots a bit
[07:36:30] <matanya>	 it won't work if you are on grub 2 or higher
[07:36:39] <matanya>	 menu.lst was replaced
[07:37:20] <andrewbogott>	 I'm sure that's the case then -- I copied that stuff blindly from our old vmbuilder setup (which /did/ use grub 1)
[07:38:02] <matanya>	 https://help.ubuntu.com/community/Grub2
[07:38:47] <andrewbogott>	 Do you know how to fix the meny properly?
[07:39:34] <matanya>	 if i understand what you tried to do there, is inserting the console into grub menu ?
[07:40:16] <andrewbogott>	 Oh, maybe?  I honestly don't know what it's trying to do, I should probably just scrap it.
[07:40:23] <andrewbogott>	 Although I guess adding a console boot option might be nice for debugging.
[07:41:16] <matanya>	 ['chroot', '{root}', 'sed', '-i', '/^kernel/s/$/ console=ttyS0/', '/boot/grub/menu.lst']
[07:41:37] <matanya>	 this looks to me like sawpping a kernel menu with a console menu
[07:43:06] <matanya>	 and now i verified jessie ships with grub2
[07:43:19] <matanya>	 so this clearly won't work
[07:45:37] <matanya>	 andrewbogott: the right way to do this would something like:
[07:45:50] <matanya>	 adding an entry in:  /etc/default/grub 
[07:45:56] <matanya>	 GRUB_CMDLINE_LINUX='console=tty0 console=ttyS0,115200n8'
[07:45:56] <matanya>	 GRUB_TERMINAL=serial
[07:45:56] <matanya>	 GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
[07:46:04] <matanya>	 and run  grub2-mkconfig -o /boot/grub2/grub.cfg 
[07:46:29] <paravoid>	 we do this in puppet
[07:46:37] <paravoid>	 I changed this to augeas recently too, for >= jessie systems
[07:47:01] <matanya>	 oh, master is here
[07:47:12] <matanya>	 andrewbogott: you don't need me anymore :)
[07:47:40] <andrewbogott>	 matanya: yes I do, you should just move on to the next obvious problem!
[07:47:59] <andrewbogott>	 matanya: are in you interested in coding on bootstrap-vz?  Trying to get btrfs or lvm working?
[07:48:18] <matanya>	 interested, yes. have time, not really :/
[07:48:27] <andrewbogott>	 ok
[07:48:40] <andrewbogott>	 Well, you may have saved me time anyway, since setting up btrfs is probably easier than lvm.
[07:48:50] <matanya>	 it sure is
[07:49:32] <matanya>	 andrewbogott: what is in /root/install_sudo.sh ?
[07:49:47] <andrewbogott>	 matanya: I believe that file is in that same patch.
[07:50:07] <andrewbogott>	 You can't just apt-get install ldap-sudo without some hoop-jumping.  Hence, a script.
[07:50:15] <andrewbogott>	 This may also be entirely unneeded these days.
[07:55:51] <matanya>	 andrewbogott: yes, tested. #apt-get install -y sudo-ldap works
[07:56:00] <matanya>	 it removes sudo, though
[07:56:06] <andrewbogott>	 matanya: that works if there's a root passwd
[07:56:09] <andrewbogott>	 it errors out if not
[07:56:43] <matanya>	 but you have one, you set it up just two lines above
[07:59:04] <andrewbogott>	 -ld?
[07:59:53] <matanya>	 or am i missing something ?
[08:00:39] <andrewbogott>	 -ld clears the password.  Or, effectively disables it.
[08:00:44] <matanya>	 and somemore hints, you are using sysvinit commands in firstboot.sh
[08:01:24] <matanya>	 yes andrewbogott, that is what i was refering to, why are you disabling the password ?
[08:02:15] <andrewbogott>	 matanya: we always disable all passwords for labs instances.  ssh only.
[08:02:28] <andrewbogott>	 Might be we could disable it /after/ installing sudo-ldap...
[08:02:46] <andrewbogott>	 but, anyway, I think that part of the script is working.  I'll look again next run.
[08:02:54] <andrewbogott>	 Tell me more about sysvinit?
[08:03:01] <matanya>	 where is the need for that, since you only allow keys ?
[08:03:16] <matanya>	 e.g: /etc/init.d/nslcd restart
[08:03:27] <matanya>	 and /etc/init.d/ssh stop
[08:03:27] <andrewbogott>	 what's systemd-speak for that?
[08:03:38] <matanya>	 systemctl restart
[08:03:54] <andrewbogott>	 systemctl restart nslcd?  Or systemclt nslcd restart?  or...
[08:04:10] <matanya>	 same as in line 7
[08:04:13] <matanya>	 systemctl restart rsyslog.service
[08:05:06] <matanya>	 it might work though, if you have systemd_sysvinit_compat, but i wouldn't go in that direction
[08:06:27] <andrewbogott>	 updated -- is that better?
[08:07:23] <matanya>	 yes
[08:07:25] <andrewbogott>	 OK, regarding the root password… I'm clearly missing something fundamental.  It looks to me like the manifest disables the root password (which is enabled previously due to the behavior of bootstrap-vz), and then installs sudo-ldap.
[08:07:31] <matanya>	 andrewbogott: salt as well
[08:07:33] <andrewbogott>	 So, what's inconsistent about that?
[08:08:23] <matanya>	 andrewbogott: i'm suggesting you disable root password as the last command, not first
[08:08:35] <andrewbogott>	 oh, sure.  That's reasonable.
[08:08:53] <matanya>	 and remove the script
[08:08:55] <andrewbogott>	 I think either way we need a script to install ldap, though, unless we implement '-y' as an option in the bootstrap manifest.
[08:09:32] <matanya>	 echo yes ?
[08:09:42] <matanya>	 or pipe y
[08:09:49] <andrewbogott>	 oh, I meant...
[08:10:02] <andrewbogott>	 well, I was thinking of moving it to the packages section.
[08:10:12] <matanya>	 even better
[08:10:17] <andrewbogott>	 If we install it via a custom command, sure, there are many options.
[08:10:35] <andrewbogott>	 What I mean is -- I thought /you/ wanted me to move it to the packages section.  Which won't work because of the 'y'
[08:11:02] <matanya>	 so how the rest work ?
[08:11:09] <matanya>	 they don't need y ?
[08:11:16] <andrewbogott>	 I presume not
[08:11:22] <matanya>	 i doubt it
[08:11:33] <andrewbogott>	 I just know that installing sudo tools is always twitchy.
[08:11:47] <andrewbogott>	 And because running a full test takes hours, I'm reluctant to prune code that is working :)
[08:12:14] <andrewbogott>	 Once I have an image that actually builds and runs I will worry about such tidying
[08:12:45] <matanya>	 if you want to be on the safe side, you can remove sudo, and then sudo-ldap installs cleanly
[08:13:02] <matanya>	 but yeah, tiding later
[08:13:26] <wikibugs>	 3Wikimedia-Labs-wikitech-interface: Many "(Cannot contact the database server: Too many connections (208.80.154.18))" errors - https://phabricator.wikimedia.org/T78492#847326 (10yuvipanda) All the Hiera hits are from the puppetmaster
[08:13:52] <matanya>	 what else andrewbogott ?
[08:13:58] <andrewbogott>	 Hm...
[08:14:24] <andrewbogott>	 Last time I tested (which, I'm in the process of setting up a new one, with code from a couple hours ago), I got one working VM (size 'medium') and one non-working VM (size 'small').
[08:14:40] <andrewbogott>	 If you stick around for 15, I'll have new logs for you to look at, we can see what's going wrong with the small one
[08:14:41] <andrewbogott>	 if it fails again
[08:15:09] <matanya>	 sure, poke me when you need. i'll go to a short breakfest 
[08:15:23] <YuviPanda>	 andrewbogott: ^^ most of those requests you pasted seem to come from the puppetmaster
[08:15:44] <andrewbogott>	 YuviPanda: yeah, I think it was a combination of things.
[08:15:58] <andrewbogott>	 there was definitely a crawl happening (you can see an entry in the paste from yahoo slurp)
[08:16:03] <andrewbogott>	 but also there may've been a puppet storm
[08:16:35] <YuviPanda>	 there was definitely a puppet storm, but that ended a while ago
[08:16:40] <YuviPanda>	 as in a few days ago
[08:17:03] <andrewbogott>	 YuviPanda: if you want to look in the access log on virt1000, maybe you'll see something more obvious to blame?
[08:17:10] <YuviPanda>	 yeah, let me look
[08:19:39] <andrewbogott>	 thanks
[08:24:28] <YuviPanda>	 andrewbogott: I don’t actually think we have a botstorm. Don’t actually have that many requests coming in
[08:24:41] <andrewbogott>	 YuviPanda: you mean, now?  Or during the outage?
[08:24:54] <YuviPanda>	 oh, now. I’m not sure when the outage was.
[08:25:02] <YuviPanda>	 outage seemed intermittent over the last few days.
[08:25:24] <andrewbogott>	 When I made that last comment on the bug about the crawler… that was in the middle of an outage.
[08:25:32] <andrewbogott>	 about 5 minute's worth.
[08:26:30] <YuviPanda>	 oh, right
[08:26:36] <YuviPanda>	 maybe puppet storms cause outages
[08:26:57] <YuviPanda>	 andrewbogott: did it subside by itself or did you have to restart mysql?
[08:27:04] <andrewbogott>	 by itself
[08:28:29] <YuviPanda>	 hmm, ok
[08:28:36] <YuviPanda>	 I”ll keep an eye out if it happens again
[08:46:25] <andrewbogott>	 matanya: ok, bootstrap-vz-small and bootstrap-vz-med.  As before, one works and one doesn't, although it's the other way 'round this time.  So clearly a race of some kind.
[08:46:30] <andrewbogott>	 You should be able to log into -small.
[08:46:48] <andrewbogott>	 And, the first boot logs are visible on wikitech, project 'openstack'.
[09:38:54] <andrewbogott>	 matanya: Stepping away for a while.  Feel free to mess with those instances as much as you like.
[09:40:39] <andrewbogott>	 meanwhile, I'm building yet another base image...
[09:41:47] <YuviPanda>	 andrewbogott: also, any news on the new hosts?
[09:42:02] <andrewbogott>	 YuviPanda: lots!  But they're still broken.
[11:16:13] <wikibugs>	 3Wikimedia-Labs-wikitech-interface: Many "(Cannot contact the database server: Too many connections (208.80.154.18))" errors - https://phabricator.wikimedia.org/T78492#847630 (10scfc) … and they come from the Puppet extension, so that's fine.  I think the problem lies in the interface MediaWiki/MySQL.  7 hits/11...
[12:25:14] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[12:51:37] <andrewbogott>	 matanya: I'm back for a few minutes… any thoughts?
[12:51:53] <andrewbogott>	 (If not, I'll build fresh VMs with the image that includes your suggested cleanup)
[12:55:10] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:10:43] <wikibugs>	 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#847840 (10Aklapper) p:5Triage>3High
[14:17:54] <wikibugs>	 3Tool-Labs-tools-tsreports: Clean up old toolserver.org references - https://phabricator.wikimedia.org/T67832#847970 (10valhallasw) a:5valhallasw>3None
[14:17:55] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-63 Query errors are not shown - https://phabricator.wikimedia.org/T60829#847973 (10valhallasw) a:5valhallasw>3None
[14:17:57] <wikibugs>	 3Tool-Labs-tools-tsreports: Possible wikidata merges - https://phabricator.wikimedia.org/T63881#847971 (10valhallasw) a:5valhallasw>3None
[14:17:58] <wikibugs>	 3Tool-Labs-tools-tsreports: nlwiki: list of all anon talk pages with a last edit > 3 years ago - https://phabricator.wikimedia.org/T63800#847972 (10valhallasw) a:5valhallasw>3None
[14:18:00] <wikibugs>	 3Tool-Labs-tools-tsreports: catswithoutinterwiki: work around wikidata caching - https://phabricator.wikimedia.org/T61960#847974 (10valhallasw) a:5valhallasw>3None
[14:18:01] <wikibugs>	 3Tool-Labs-tools-tsreports: Allow filtering of the final report - https://phabricator.wikimedia.org/T61764#847976 (10valhallasw) a:5valhallasw>3None
[14:18:03] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-2 should be possible to translate field titles - https://phabricator.wikimedia.org/T60845#847978 (10valhallasw) a:5valhallasw>3None
[14:18:04] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-9 Document machine-readable report output - https://phabricator.wikimedia.org/T60842#847975 (10valhallasw) a:5valhallasw>3None
[14:18:05] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-1 Italian translation for Reports - https://phabricator.wikimedia.org/T60846#847977 (10valhallasw) a:5valhallasw>3None
[14:18:06] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-7 Add a thumbnail field - https://phabricator.wikimedia.org/T60844#847979 (10valhallasw) a:5valhallasw>3None
[14:18:08] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-20 Use translatewiki for message translations - https://phabricator.wikimedia.org/T60840#847982 (10valhallasw) a:5valhallasw>3None
[14:18:09] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-16 Show the replication lag on the results page - https://phabricator.wikimedia.org/T60841#847981 (10valhallasw) a:5valhallasw>3None
[14:18:10] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-8 Add a variable type for page name - https://phabricator.wikimedia.org/T60843#847980 (10valhallasw) a:5valhallasw>3None
[14:18:11] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-35 Add a pager for long reports - https://phabricator.wikimedia.org/T60838#847984 (10valhallasw) a:5valhallasw>3None
[14:18:12] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-35 Add a pager for long reports - https://phabricator.wikimedia.org/T60839#847983 (10valhallasw) a:5valhallasw>3None
[14:18:14] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-47 create a css class - https://phabricator.wikimedia.org/T60837#847985 (10valhallasw) a:5valhallasw>3None
[14:18:15] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-48 Tables should have a different color on hover - https://phabricator.wikimedia.org/T60836#847986 (10valhallasw) a:5valhallasw>3None
[14:18:16] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-52 Add option to output report results to a wiki page - https://phabricator.wikimedia.org/T60833#847989 (10valhallasw) a:5valhallasw>3None
[14:18:17] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-49 IP block SQL Query on en-wiki - https://phabricator.wikimedia.org/T60835#847987 (10valhallasw) a:5valhallasw>3None
[14:18:18] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-51 Add history function to reports - https://phabricator.wikimedia.org/T60834#847988 (10valhallasw) a:5valhallasw>3None
[14:18:20] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-62 RSS output - https://phabricator.wikimedia.org/T60830#847992 (10valhallasw) a:5valhallasw>3None
[14:18:21] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-60 webstats are broken - https://phabricator.wikimedia.org/T60831#847991 (10valhallasw) a:5valhallasw>3None
[14:18:22] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-57 Create API documentation - https://phabricator.wikimedia.org/T60832#847990 (10valhallasw) a:5valhallasw>3None
[14:18:23] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-71 1226, "User 'reports' has exceeded the 'max_user_connections' resource (current value: 15)" due to worker processes not closing their connection to the user database server - https://phabricator.wikimedia.org/T60825#847994 (10valhallasw) a:5valhallasw>3None
[14:18:24] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-68 Implement dynamic cache duration - https://phabricator.wikimedia.org/T60826#847993 (10valhallasw) a:5valhallasw>3None
[14:18:25] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-73 Show a list of recently finished queries (possibly also via RSS) - https://phabricator.wikimedia.org/T60823#847995 (10valhallasw) a:5valhallasw>3None
[14:18:46] <YuviPanda|brb>	 valhallasw`cloud: heh, not going to maintain it anymore? :)
[14:19:08] <valhallasw`cloud>	 YuviPanda|brb: well, at least not giving the impression I'm going to pick up those tasks soon :-p
[14:19:18] <YuviPanda|brb>	 heh
[14:19:25] <YuviPanda|brb>	 I should really make Quarry subsume these soon
[14:19:41] <valhallasw`cloud>	 but yeah, I'd like to merge with Quarry as much as possible, really.
[14:19:51] <valhallasw`cloud>	 mainly because the tsreports code is teh horror, mostly
[14:19:56] <valhallasw`cloud>	 pre-flask WSGI
[14:19:57] <valhallasw`cloud>	 *brr*
[14:20:33] <YuviPanda|brb>	 valhallasw`cloud: oh wow
[14:20:44] <YuviPanda|brb>	 valhallasw`cloud: I wonder if I added some way to make wikilinks + canonical namespace names...
[14:21:15] <YuviPanda|brb>	 anyway, off now, brb
[14:38:12] <valhallasw`cloud>	 YuviPanda|brb: I'll make a note to myself to create some bugs for 'tsreports parity'
[14:41:06] <wikibugs>	 3Quarry, Tool-Labs-tools-tsreports: Quarry-TSreports feature parity - https://phabricator.wikimedia.org/T78549#848037 (10valhallasw) 3NEW
[18:16:25] <wikibugs>	 3Tool-Labs: SQL server is not responding .. responsive - https://phabricator.wikimedia.org/T76930#848392 (10Aklapper) @GerardM: What are the exact steps to reproduce this problem?
[18:33:00] <wikibugs>	 3Tool-Labs: SQL server is not responding .. responsive - https://phabricator.wikimedia.org/T76930#848468 (10GerardM) Hoi, I create a ticket when this happens. Typically it is something that is wrong at that time... I suffer from it and it does not go away with rebooting my system or clearing the cache.  Some iss...
[19:18:48] <YuviPanda>	 valhallasw`cloud: around?
[19:20:13] <valhallasw`cloud>	 YuviPanda: ta
[19:20:36] <YuviPanda>	 valhallasw`cloud: so… i was thinking of adding ‘Quarry Pads’, which will be associated with a query, and run a template of some sort. only output will be visible
[19:20:40] <YuviPanda>	 similar to tsreports
[19:21:10] <valhallasw`cloud>	 yes, that was something going around my brain, too.
[19:21:38] <YuviPanda>	 valhallasw`cloud: so from a query, you just click ‘create pad’ and it shows up a way for you to configure how to view it.
[19:22:00] <YuviPanda>	 valhallasw`cloud: the other option is to make pads be associated with any number of queries
[19:22:55] <valhallasw`cloud>	 nah, just a simple configuration mapper is better, I think.
[19:23:14] <YuviPanda>	 yeah, but I suspect the same pad (outputconfig) can be used for multiple queries easily
[19:23:36] <valhallasw`cloud>	 YuviPanda: from my tsreports experience I can tell you: not as often as you'd hope
[19:23:41] <valhallasw`cloud>	 often it's slightly different
[19:23:52] <valhallasw`cloud>	 and it's dead simple to build a new one anyway
[19:23:54] <YuviPanda>	 true, but quarry already has like, a few thousand queries...
[19:23:59] <YuviPanda>	 true, but they drift...
[19:24:07] <valhallasw`cloud>	 ?
[19:24:08] <YuviPanda>	 also I’m not sure how exactly to make the ‘run on any wiki’ possible
[19:24:25] <valhallasw`cloud>	 just add use dbname; in front of the query
[19:24:27] <YuviPanda>	 valhallasw`cloud: think of this as templates on wiki. Wouldn’t it have been wonderful if they were on a central wiki from the start?
[19:24:30] <valhallasw`cloud>	 if it breaks whatever.
[19:24:50] <valhallasw`cloud>	 YuviPanda: I don't think that comparison works.
[19:24:58] <YuviPanda>	 why not?
[19:25:05] <valhallasw`cloud>	 YuviPanda: you don't have 100s of queries where you want the exact same output
[19:25:10] <valhallasw`cloud>	 you want templates for *fields*
[19:25:12] <valhallasw`cloud>	 not for queries
[19:25:18] <YuviPanda>	 hmmm
[19:25:21] <valhallasw`cloud>	 so a 'page' field and an 'image preview' field
[19:25:30] <YuviPanda>	 right
[19:25:37] <YuviPanda>	 so if this is jinja2 that should just be custom filters
[19:25:41] <YuviPanda>	 which should be trivial
[19:25:48] <valhallasw`cloud>	 I suppose, yes.
[19:26:19] <valhallasw`cloud>	 YuviPanda: ahh, you're going for full jinja support
[19:26:25] <YuviPanda>	 valhallasw`cloud: yes.
[19:26:29] <valhallasw`cloud>	 then indeed some sort of default 'table' template would be useful, yeah
[19:26:34] <YuviPanda>	 with helpers for normal common layouts
[19:26:44] <YuviPanda>	 yeah, table, ‘summary’ and stuff like that
[19:26:49] <valhallasw`cloud>	 sounds good
[19:27:22] <YuviPanda>	 valhallasw`cloud: still not sure how exactly to avoid XSS
[19:27:30] <valhallasw`cloud>	 YuviPanda: XSS?
[19:27:42] <valhallasw`cloud>	 as in 'someone dumps html in the jinja template to steal cookies'?
[19:27:43] <YuviPanda>	 avoid people making jinja2 output random <script> tags :)
[19:27:44] <YuviPanda>	 yeah
[19:27:46] <YuviPanda>	 that
[19:28:04] <valhallasw`cloud>	 YuviPanda: we can do OAuth style stuff
[19:28:13] <valhallasw`cloud>	 or...
[19:28:15] <YuviPanda>	 hmm?
[19:28:22] <valhallasw`cloud>	 as in: it has to be OK'ed
[19:28:26] <valhallasw`cloud>	 or a seperate subdomain?
[19:28:35] <YuviPanda>	 hmmm
[19:28:36] <valhallasw`cloud>	 quarry-reports.wmflabs.org where no-one actually logs in
[19:28:40] <YuviPanda>	 XSS is ugly no mater what.
[19:28:44] <YuviPanda>	 no?
[19:29:00] <YuviPanda>	 on the other hand, *allowing* JS would probably open a wonderful/terrible world on its own
[19:29:15] <valhallasw`cloud>	 I mean, you can try an HTML stripper
[19:29:24] <YuviPanda>	 tables, etc also are HTML
[19:29:26] <valhallasw`cloud>	 or throw it through a mediawiki parser
[19:29:29] <valhallasw`cloud>	 ta
[19:29:30] <YuviPanda>	 ohgodno
[19:29:33] <YuviPanda>	 noooo
[19:29:37] <YuviPanda>	 actually
[19:29:39] <YuviPanda>	 hmm
[19:29:41] <YuviPanda>	 that’s not a bad idea
[19:29:43] * YuviPanda cringes
[19:29:46] <valhallasw`cloud>	 mediawiki parser in the context of the site where it runs?
[19:29:47] <YuviPanda>	 but will need to support templates
[19:29:56] <valhallasw`cloud>	 just send it to the right api.php to parse
[19:30:01] <valhallasw`cloud>	 and wikilinks will work automagically
[19:30:07] <valhallasw`cloud>	 OH GOD THIS IS EVIL
[19:30:11] <YuviPanda>	 YES
[19:30:13] <YuviPanda>	 VERY VERY EVIL
[19:30:38] <YuviPanda>	 valhallasw`cloud: and you can do tables, and use templates on that wiki
[19:31:38] <valhallasw`cloud>	 yep.
[19:31:49] <valhallasw`cloud>	 might as well just dump the data there :p
[19:32:05] <valhallasw`cloud>	 with a button 'REQUEST UPDATE'
[19:33:06] <YuviPanda>	 valhallasw`cloud: noooooooo
[19:33:11] <YuviPanda>	 valhallasw`cloud: that’s just so many levels of evil
[19:35:08] <YuviPanda>	 valhallasw`cloud: NOOOO
[19:35:13] <YuviPanda>	 valhallasw`cloud: let’s not do that
[19:35:26] <YuviPanda>	 it would make a lot of things easy...
[19:35:44] <YuviPanda>	 valhallasw`cloud: have jinja2 generate wikitext, and put it through api=parse
[19:35:46] <YuviPanda>	 and store result
[19:35:53] <YuviPanda>	 genius, but evil
[19:36:15] <valhallasw`cloud>	 yep.
[19:36:56] <YuviPanda>	 valhallasw`cloud: so tempting
[19:37:41] <valhallasw`cloud>	 let's do it.
[19:38:08] <YuviPanda>	 valhallasw`cloud: we must resist that temptation
[19:38:15] <YuviPanda>	 valhallasw`cloud: it shall bring nothing but bad fortune and evil
[19:38:52] <valhallasw`cloud>	 YuviPanda: why? it makes every part of the machine do it's best part
[19:39:43] <YuviPanda>	 valhallasw`cloud: but wikitext sucks
[19:40:14] <valhallasw`cloud>	 Yes. On the other hand, people know wikitext, and it automatically prevents all kinds of nastyness
[19:41:45] <valhallasw`cloud>	 YuviPanda: I mean, you either need to use magic html fixing (which often fails) or you get some text-based formatting language
[19:41:50] <valhallasw`cloud>	 might as well be wikitext then
[19:42:05] <YuviPanda>	 valhallasw`cloud: or we could go full hog, and allow arbitrary HTML and JS and CSS
[19:42:12] <YuviPanda>	 and put it on a different domain
[19:42:26] <valhallasw`cloud>	 this is also tempting
[19:42:35] <valhallasw`cloud>	 but I'm a bit afraid of all the popup mess that's going to give
[19:42:45] <valhallasw`cloud>	 isn't there a no-execute header or something
[19:43:30] <valhallasw`cloud>	 yeah, Content-Security-Policy
[19:45:33] <YuviPanda>	 valhallasw`cloud: popup mess?
[19:45:44] <YuviPanda>	 valhallasw`cloud: I’ll provide jQuery and Bootstrap by default
[19:45:48] <valhallasw`cloud>	 YuviPanda: alert()alert()alert()
[19:46:01] <valhallasw`cloud>	 YuviPanda: why jQ? :/
[19:46:26] <YuviPanda>	 valhallasw`cloud: so you could write whatever JS you want :)
[19:46:37] <valhallasw`cloud>	 YuviPanda: I'm not so sure about the JS :P
[19:46:57] <YuviPanda>	 valhallasw`cloud: hehe :)
[19:47:19] <valhallasw`cloud>	 I was going to suggest freeform HTML + CSS and then block JS using Content-Security-Policy
[19:47:25] <valhallasw`cloud>	 but that doesn't save older browsers
[19:47:26] <valhallasw`cloud>	 :/
[19:47:36] <YuviPanda>	 indeed
[19:47:41] <YuviPanda>	 or wikitext + CSS
[19:48:00] <valhallasw`cloud>	 let's just do wikitext with some standard quarry css
[19:48:05] <valhallasw`cloud>	 wikitext allows some inline CSS
[19:48:16] <YuviPanda>	 yeah, but inline CSS makes things boil
[19:48:47] <valhallasw`cloud>	 mmm add a CSS field that will be embedded in the html, I guess.
[19:49:38] <YuviPanda>	 yeah
[19:52:41] <valhallasw`cloud>	 it's cold here
[19:52:45] <valhallasw`cloud>	 time for tea!
[19:53:03] <YuviPanda>	 valhallasw`cloud: heh :)
[20:48:11] <wikibugs>	 3Tool-Labs-tools-tsreports, Quarry: RSS feeds - https://phabricator.wikimedia.org/T60830#848877 (10valhallasw)
[20:49:22] <wikibugs>	 3Quarry: Show a list of recently finished queries (possibly also via RSS) - https://phabricator.wikimedia.org/T60823#848882 (10valhallasw)
[20:49:52] <wikibugs>	 3Quarry: Create 'reports' feature - https://phabricator.wikimedia.org/T78593#848886 (10valhallasw) 3NEW
[20:50:07] <wikibugs>	 3Quarry: Show a list of recently finished queries (possibly also via RSS) - https://phabricator.wikimedia.org/T60823#601832 (10valhallasw)
[20:50:32] <wikibugs>	 3Quarry: Create 'reports' feature - https://phabricator.wikimedia.org/T78593#848886 (10valhallasw)
[20:50:53] <wikibugs>	 3Tool-Labs-tools-tsreports, Quarry: Quarry-TSreports feature parity - https://phabricator.wikimedia.org/T78549#848902 (10valhallasw)
[20:51:14] <YuviPanda>	 valhallasw`cloud: so, I create a ‘reports’ feature on quarry, which lets you use jinja2 to output wikitext/<other-markup-language>, and then add the tsreportsish caching method
[20:51:51] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-60 webstats are broken - https://phabricator.wikimedia.org/T60831#848903 (10valhallasw) 5Open>3Invalid a:3valhallasw
[20:52:15] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-68 Implement dynamic cache duration - https://phabricator.wikimedia.org/T60826#848906 (10valhallasw)
[20:52:31] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-71 1226, "User 'reports' has exceeded the 'max_user_connections' resource (current value: 15)" due to worker processes not closing their connection to the user database server - https://phabricator.wikimedia.org/T60825#848907 (10valhallasw) 5Open>3Invalid a:3valhallasw N...
[20:53:09] <wikibugs>	 3Tool-Labs-tools-tsreports, Quarry: Provide redirects from old tsreports reports to new quarry reports - https://phabricator.wikimedia.org/T78595#848910 (10valhallasw) 3NEW
[20:53:51] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-52 Add option to output report results to a wiki page - https://phabricator.wikimedia.org/T60833#848917 (10valhallasw)
[20:54:12] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-57 Create API documentation - https://phabricator.wikimedia.org/T60832#848926 (10valhallasw) 5Open>3declined a:3valhallasw
[20:54:34] <YuviPanda>	 valhallasw`cloud: aaugh, jinja2 and wikitext are going to have a hell of a time mixing
[20:54:40] <YuviPanda>	 because of the default {{ in jinja2
[20:54:42] <valhallasw`cloud>	 maybe, yeah
[20:55:10] <YuviPanda>	 valhallasw`cloud: LET’S WRITE IT IN RUBY AND MAKE IT USE ERB!
[20:55:41] * valhallasw`cloud hits YuviPanda
[20:56:05] <wikibugs>	 3Quarry: Ideas for reports - https://phabricator.wikimedia.org/T78597#848934 (10valhallasw) 3NEW
[20:56:19] <YuviPanda>	 valhallasw`cloud: thanks, that seems to have fixed it
[20:56:33] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-49 IP block SQL Query on en-wiki - https://phabricator.wikimedia.org/T60835#848941 (10valhallasw)
[20:56:43] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-48 Tables should have a different color on hover - https://phabricator.wikimedia.org/T60836#848942 (10valhallasw) 5Open>3declined a:3valhallasw
[20:57:11] <wikibugs>	 3Quarry: REPORTS-49 IP block SQL Query on en-wiki - https://phabricator.wikimedia.org/T60835#602808 (10valhallasw)
[20:59:44] <wikibugs>	 3Quarry: REPORTS-52 Add option to output report results to a wiki page - https://phabricator.wikimedia.org/T60833#848947 (10valhallasw)
[20:59:56] <valhallasw`cloud>	 YuviPanda: we can also just go for table + header for now
[21:00:21] <wikibugs>	 3Quarry: REPORTS-68 Implement dynamic cache duration - https://phabricator.wikimedia.org/T60826#848950 (10valhallasw)
[21:00:28] <YuviPanda>	 valhallasw`cloud: well, if I’m doing it once, might as well do it right
[21:00:29] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-51 Add history function to reports - https://phabricator.wikimedia.org/T60834#848952 (10valhallasw) 5Open>3declined a:3valhallasw
[21:00:51] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-35 Add a pager for long reports - https://phabricator.wikimedia.org/T60839#848954 (10valhallasw)
[21:00:59] <wikibugs>	 3Quarry: REPORTS-35 Add a pager for long reports - https://phabricator.wikimedia.org/T60839#603131 (10valhallasw)
[21:01:11] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-20 Use translatewiki for message translations - https://phabricator.wikimedia.org/T60840#848957 (10valhallasw) 5Open>3Resolved a:3valhallasw
[21:01:29] <YuviPanda>	 valhallasw`cloud: also quarry isn’t translated
[21:01:52] <wikibugs>	 3Quarry: REPORTS-35 Add a pager for long reports - https://phabricator.wikimedia.org/T60839#848960 (10yuvipanda) Quarry already has a pager.
[21:02:01] <valhallasw`cloud>	 also for reports? :p
[21:02:11] <wikibugs>	 3Quarry: REPORTS-35 Add a pager for long reports - https://phabricator.wikimedia.org/T60839#848962 (10valhallasw) 5Open>3declined a:3valhallasw
[21:02:13] <valhallasw`cloud>	 well, whatever, we'll see
[21:02:28] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-47 create a css class - https://phabricator.wikimedia.org/T60837#848969 (10valhallasw) 5Open>3declined a:3valhallasw
[21:02:45] <YuviPanda>	 valhallasw`cloud: aaah, yeah. well, I suppose the table code there will just re-use the current table code...
[21:03:06] <wikibugs>	 3Quarry: REPORTS-35 Add a pager for long reports - https://phabricator.wikimedia.org/T60839#603131 (10valhallasw)
[21:03:08] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-35 Add a pager for long reports - https://phabricator.wikimedia.org/T60838#848972 (10valhallasw)
[21:03:09] <YuviPanda>	 valhallasw`cloud: shit, just realized pagination for reports will be complicated
[21:03:28] <valhallasw`cloud>	 YuviPanda: meh, maybe.
[21:03:47] <valhallasw`cloud>	 just let the template author figure it out :-p
[21:04:03] <YuviPanda>	 tch tch :)
[21:04:31] <wikibugs>	 3Quarry: Show replication lag - https://phabricator.wikimedia.org/T60841#848974 (10valhallasw)
[21:04:50] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-1 Italian translation for Reports - https://phabricator.wikimedia.org/T60846#848977 (10valhallasw) 5Open>3declined a:3valhallasw Now via twn.
[21:05:09] <wikibugs>	 3Quarry: Show replication lag - https://phabricator.wikimedia.org/T60841#848980 (10yuvipanda) But labsdb always has zero replag! :D
[21:05:12] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-2 should be possible to translate field titles - https://phabricator.wikimedia.org/T60845#848982 (10valhallasw) 5Open>3declined a:3valhallasw
[21:05:26] <wikibugs>	 3Tool-Labs-tools-tsreports: Allow filtering of the final report - https://phabricator.wikimedia.org/T61764#848984 (10valhallasw)
[21:05:35] <wikibugs>	 3Quarry: Allow filtering of the final report - https://phabricator.wikimedia.org/T61764#616142 (10valhallasw)
[21:06:12] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-8 Add a variable type for page name - https://phabricator.wikimedia.org/T60843#848987 (10valhallasw) 5Open>3declined a:3valhallasw User-defined parameters are disabled.
[21:06:25] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-7 Add a thumbnail field - https://phabricator.wikimedia.org/T60844#848993 (10valhallasw)
[21:06:33] <Krinkle>	 YuviPanda: Could you maybe help investigate what happened to integration-slave1001?
[21:06:37] <wikibugs>	 3Quarry: Reports: Add a thumbnail field - https://phabricator.wikimedia.org/T60844#848994 (10valhallasw)
[21:06:48] <Krinkle>	 YuviPanda: See https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL for entries related to it
[21:06:52] <Krinkle>	 dating about 30 days back
[21:06:52] <YuviPanda>	 Krinkle: oh, what happened?
[21:06:56] <Krinkle>	 It's dead
[21:06:59] <YuviPanda>	 looking
[21:07:09] <wikibugs>	 3Tool-Labs-tools-tsreports: Possible wikidata merges - https://phabricator.wikimedia.org/T63881#848997 (10valhallasw)
[21:07:09] <YuviPanda>	 oh, hmm
[21:07:10] <Krinkle>	 but it's not urgent. I'd rather you figure out what happened then fix it.
[21:07:18] <wikibugs>	 3Quarry: Possible wikidata merges - https://phabricator.wikimedia.org/T63881#651842 (10valhallasw)
[21:07:20] <Krinkle>	 Seems more valuable
[21:07:26] <YuviPanda>	 Krinkle: does it say ‘SHUTOFF’ in the wikitech interface?
[21:07:40] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-9 Document machine-readable report output - https://phabricator.wikimedia.org/T60842#849002 (10valhallasw) 5Open>3declined a:3valhallasw Let Quarry handle it.
[21:07:56] <wikibugs>	 3Tool-Labs-tools-tsreports: Clean up old toolserver.org references - https://phabricator.wikimedia.org/T67832#849005 (10valhallasw) p:5Triage>3Normal
[21:08:16] <Krinkle>	 YuviPanda: Where? I rebooted it Dec 5 for a second time to try and restore it, but to no avail
[21:08:18] <Krinkle>	 https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource%3AI-000001bd.eqiad.wmflabs&diff=137008&oldid=137007
[21:08:40] <Krinkle>	 Dec 2 05:26 Krinkle: integration-slave1001 has been down since the failed reboot on 28 November 2014. Still unreachable over ssh and no Jenkins slave agent. 
[21:08:48] <wikibugs>	 3Tool-Labs-tools-tsreports: catswithoutinterwiki: work around wikidata caching - https://phabricator.wikimedia.org/T61960#849012 (10valhallasw) 5Open>3declined a:3valhallasw I think writing a better query is a better option; this should be done in Quarry.
[21:09:04] <wikibugs>	 3Tool-Labs-tools-tsreports: nlwiki: list of all anon talk pages with a last edit > 3 years ago - https://phabricator.wikimedia.org/T63800#849016 (10valhallasw)
[21:09:15] <wikibugs>	 3Quarry: Report: nlwiki: list of all anon talk pages with a last edit > 3 years ago - https://phabricator.wikimedia.org/T63800#849017 (10valhallasw)
[21:09:38] <wikibugs>	 3Tool-Labs-tools-tsreports: REPORTS-63 Query errors are not shown - https://phabricator.wikimedia.org/T60829#849020 (10valhallasw) 5Open>3declined a:3valhallasw I think Quarry does show errors.
[21:09:43] <Krinkle>	 YuviPanda: https://integration.wikimedia.org/ci/computer/integration-slave1001/log shows: https://gist.github.com/Krinkle/82d612b002fc9533bda7
[21:09:51] <valhallasw`cloud>	 ok, spam is done!
[21:11:32] <YuviPanda>	 valhallasw`cloud: you should join -research
[21:13:02] <YuviPanda>	 Krinkle: hmm, doesn’t work with my root key either, and the stupid bug causing it to show up empty is in wikitech again
[21:14:00] <YuviPanda>	 uh oh, and a relogin doesn’t fix it
[21:14:02] * YuviPanda swears a bit
[21:18:18] <YuviPanda>	 Krinkle: https://dpaste.de/0VEP is the output on console
[21:18:22] <YuviPanda>	 (I can obtain that from virt1000)
[21:26:53] <YuviPanda>	 Krinkle: so it… could possibly be dead, or I could get an interactive console and try to bypass the mount error 
[21:27:03] <YuviPanda>	 but libvirt doesn’t seem to be configured properly, so I’d have to wait for andrewbogott_afk 
[21:28:29] <wikibugs>	 3Tool-Labs: SQL server is not responding .. responsive - https://phabricator.wikimedia.org/T76930#849103 (10Aklapper) Alright, but my question was "What are the exact steps to reproduce this problem?".  I sit in front of a computer which has access to the internet. No idea what to do to get to "Getting WDQ data"...
[21:31:12] <Steinsplitter>	 was there recently a change with jsub? jsub has problems with dir?
[21:32:31] <Steinsplitter>	 coren?
[21:33:21] <YuviPanda>	 Steinsplitter: hi! what problems are you encountering?
[21:33:30] <wikibugs>	 3Wikimedia-Labs-Infrastructure: Internal DNS look-ups fail every once in a while - https://phabricator.wikimedia.org/T72076#849142 (10hashar) I looked at Coren original patch for labs nscd.conf https://gerrit.wikimedia.org/r/#/c/157816/1/modules/ldap/files/nscd-labs.conf  On deployment-bastion we have:  ```...
[21:34:00] <Steinsplitter>	 YuviPanda: when executing my scripts in jsub, the fail. seems to be a dir problem.
[21:34:18] <Steinsplitter>	 but the script has worked on labs for months now...
[21:34:57] <YuviPanda>	 Steinsplitter: which script? what exactly do you mean by a ‘dir’ problem?
[21:35:23] <Steinsplitter>	 directory, if i exeute it in ssh it works, but it don't with jsub
[21:36:04] <Betacommand>	 Steinsplitter: probably a working directory issue, let me get my notes
[21:36:18] <YuviPanda>	 Steinsplitter: can you tell me which script this is?
[21:36:22] <YuviPanda>	 ah, what Betacommand said as well :)
[21:37:03] <Steinsplitter>	 YuviPanda: crontabs in sbot , the Rotatebot one.
[21:37:35] <Betacommand>	 Steinsplitter: use -cwd
[21:38:16] <Betacommand>	 also make sure your using the full path and not relative path in the crontab
[21:39:14] <YuviPanda>	 Steinsplitter: I see no crontabs for rotatebot
[21:39:20] <YuviPanda>	 crontab -l lists nothing
[21:39:38] <YuviPanda>	 Steinsplitter: ah, it’s in sbot
[21:39:38] <YuviPanda>	 nevermind
[21:40:24] <Steinsplitter>	 Betacommand: adding -cwd to jsub dosen't work.
[21:40:58] <Steinsplitter>	 there schould be a way to execute crontabs with the same config as in ssh
[21:41:33] <YuviPanda>	 Steinsplitter: I see php errors in rotatebot.4.err
[21:41:51] <Steinsplitter>	 YuviPanda: yes, because of jsub
[21:42:04] <Steinsplitter>	 if i execute the script in ssh work
[21:42:49] <YuviPanda>	 Steinsplitter: hmm, it looks like a logic error to me? $vcx is defined only if user_registration is false (line 252)
[21:43:11] <YuviPanda>	 but on line 277 it seems that it is checked assuming the variable exists
[21:43:29] <Steinsplitter>	 isn't there a way to solve the problem without changing me scripts?
[21:43:38] <YuviPanda>	 err, this *is* a problem in your script?
[21:44:23] <Steinsplitter>	 but why it works in ssh... perfectly. it also works on my testspace without jsub perfectly.
[21:44:48] * Steinsplitter is crying
[21:44:58] <Steinsplitter>	 every mont breaks something because of the mw infrastructure
[21:45:10] <YuviPanda>	 Steinsplitter: are you running them with the exact same commandline? perhaps when you’re running it off ssh $user_registration isn’t set for some reason
[21:45:36] <Steinsplitter>	 yes, it is exactly the same. and i haven't changed nothing in the last days
[21:50:02] <Steinsplitter>	 YuviPanda: if i give jsub the full dir, he schould be able to fin all ~/  , like a norma crontab.
[21:50:33] <YuviPanda>	 Steinsplitter: hmm, why should that matter? are you including other PHP files?
[21:50:40] <Steinsplitter>	 yes
[21:52:38] <Dragonfly6-7>	 hi there
[21:53:26] <YuviPanda>	 Steinsplitter: are you also looking at the following lines in the .err file?
[21:53:31] <YuviPanda>	 PHP Stack trace:
[21:53:31] <YuviPanda>	 PHP   1. {main}() /data/project/sbot/Rotatebot/rotbot.php:0
[21:53:31] <YuviPanda>	 PHP Notice:  Undefined variable: vcx in /data/project/sbot/Rotatebot/rotbot.php on line 277
[21:53:44] <Steinsplitter>	 there is nothing wrong O_O
[21:53:51] <YuviPanda>	 oh wait, they’re notices.
[21:53:59] <YuviPanda>	 not actual errors
[21:54:01] <YuviPanda>	 red herring, sorry
[21:54:06] <Dragonfly6-7>	 there's a problem with the Articles Created tool
[21:54:16] <YuviPanda>	 Steinsplitter: so what do you mean by ‘does not work’ if the .err file isn’t showing any errors?
[21:54:22] <Dragonfly6-7>	 given that I can find one specific problem reproducibly, I woldn't be surprised if the problem occurs elsewhere
[21:54:38] <YuviPanda>	 Steinsplitter: I’m not saying the code is wrong, I’m just trying to understand what exactly is the problem so I can repro it
[21:54:44] <YuviPanda>	 Dragonfly6-7: oh, what specific problem?
[21:55:14] <Dragonfly6-7>	 YuviPanda - at least one article which I know I created (and which is still up) does not appear on my list of Articles Created
[21:55:15] <Krinkle>	 YuviPanda: thanks. I'll try and build a new one. Assuming we can still build new precise instances without puppet failures
[21:55:27] <Dragonfly6-7>	 https://tools.wmflabs.org/xtools/pages/index.php?user=DragonflySixtyseven&lang=en&wiki=wikipedia&namespace=0&redirects=noredirects
[21:55:41] <Steinsplitter>	 YuviPanda: executing the script with jsub dosen't work, there seems to isuisse since a few deys with the directorys.
[21:55:42] <Dragonfly6-7>	 [[Arsen Darnay]] should be included on there, but is not
[21:55:52] <YuviPanda>	 Dragonfly6-7: aaah, you’ll have to contact the tool developers, I’m afraid.
[21:55:57] <Betacommand>	 Dragonfly6-7: let me check something
[21:56:07] <YuviPanda>	 Krinkle: yeah, precise still works. just not labs-vagrant on precise.
[21:57:12] <Betacommand>	 Steinsplitter: what is the command your setting in jsub
[21:57:21] <YuviPanda>	 Steinsplitter: sorry, can you tell me what you mean by ‘does not work’?
[21:57:25] <Steinsplitter>	 jsub -N rotatebot.4 -mem 980m -once -cwd -quiet php /data/project/sbot/Rotatebot/rotbot.php
[21:57:43] <Steinsplitter>	 Yuvi: i can't debug without erroroutput. jsub dosen't execut it
[21:59:32] <YuviPanda>	 Steinsplitter: I just executed it via sub, and moved the old .err file before I did, and it executes fine?
[21:59:36] <YuviPanda>	 *jsub
[21:59:37] <Betacommand>	 Steinsplitter: try jsub -N rotatebot.4 -mem 2G -once -cwd -quiet php /data/project/sbot/Rotatebot/rotbot.php
[21:59:42] <YuviPanda>	 and there’s a new .err file with all the notices
[22:01:02] <Steinsplitter>	 Betacommand: nope
[22:02:18] <Betacommand>	 Steinsplitter: does the .out or .err contain the fatal error message?
[22:03:08] <YuviPanda>	 Steinsplitter: Betacommand  aha
[22:03:11] <YuviPanda>	 Starte Bot!
[22:03:11] <YuviPanda>	 Could not get lock. Lock file already present. Exit.
[22:03:15] <YuviPanda>	 so that’s what happens
[22:03:23] <YuviPanda>	 your lock file is stale, and so it doesn’t actually run?
[22:03:25] <YuviPanda>	 I see this in .out
[22:03:34] <Steinsplitter>	 rm the lock and try again.
[22:03:41] <Steinsplitter>	 it isn't the lockfile
[22:03:53] <YuviPanda>	 where is the lockfile?
[22:04:27] <Steinsplitter>	 in ~/Rotatebot
[22:04:55] <Steinsplitter>	 try to execute in the folder with php rotbot.php  ... it works fine. 
[22:08:41] <Steinsplitter>	 I have a live. I have a job. I do not have time to fix every monts/week things because wmf can't be bothered to provide a infrastructure that works in real life.
[22:13:08] <YuviPanda>	 Steinsplitter: hmm, so there seems to be something amiss with the locking code. I just temporarily commented it out, and now the bot works via jsub
[22:13:20] <YuviPanda>	 Steinsplitter: on running it I see that .out ends with:
[22:13:21] <YuviPanda>	 Bot finished.
[22:15:53] <Steinsplitter>	 That is strage. it seems to be that jsub calls directory differently. executing it in a normal way (in ssh simpel with php or exec) works with the lock too.  Okay, thank you then :) i go to sleep now. bye :)
[22:17:05] <YuviPanda>	 Steinsplitter: I’ll keep looking at it, but it should work for now because it’s no longer trying to lock
[22:17:12] <YuviPanda>	 Steinsplitter: and -once should make sure your job is executing only once
[22:19:00] <Steinsplitter>	 ok :), have a nice evening
[22:37:10] <Dragonfly6-7>	 Anyone?
[22:39:15] <YuviPanda>	 Dragonfly6-7: I think cyberpower is the maintainer for that tool, so he might be able to help? Betacommand might know a thing or two as well?
[22:41:45] <Krinkle>	 YuviPanda: And what's with these two again, they clog up syslog with entries like every other second.
[22:41:46] <Krinkle>	 nslcd[29117]: [a26c33] <group=50380> error writing to client: Broken pipe
[22:41:46] <Krinkle>	 nslcd[29117]: [cfb610] <group=50062> error writing to client: Broken pipe
[22:41:52] <Krinkle>	 Every instance I've ever seen.
[22:42:02] <Krinkle>	 Known issue?
[22:42:07] <YuviPanda>	 hmm, no.
[22:42:12] <YuviPanda>	 that’s the LDAP client
[22:42:19] <Krinkle>	 I've reported it to Ryan Lane, Coren and andrew over the past year.
[22:42:37] <Krinkle>	 Seems to be unimportant from what I gather, but rather annoying.
[22:42:58] <YuviPanda>	 Krinkle: interesting, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=685504
[22:43:09] <YuviPanda>	 apparently group=50380 is something that’s too huge for nslcd
[22:45:17] <Krinkle>	 integration-slave1002 nslcd[29117]: [......] <group=50380> error writing to client: Broken pipe
[22:45:18] <Krinkle>	 integration-slave1002 nslcd[29117]: [......] <group=1003> error writing to client: Broken pipe
[22:45:18] <Krinkle>	 integration-slave1002 nslcd[29117]: [......] <group=50062> error writing to client: Broken pipe
[22:45:19] <Krinkle>	 integration-slave1002 nslcd[29117]: [......] <group=550> error writing to client: Broken pipe
[22:45:22] <Krinkle>	 These are the most common ones
[22:45:26] <Krinkle>	 (I normalised it and did a unique)
[22:45:33] <YuviPanda>	 let me look at what those groups are
[22:45:34] <Krinkle>	 from the past 24h
[22:45:36] <Krinkle>	 Thx
[22:45:45] <Krinkle>	 YuviPanda: I'll file a phab if you want.
[22:45:51] <YuviPanda>	 Krinkle: yesplease!
[22:47:26] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Labs-Team: Fix syslog error "nslcd[29117]: error writing to client: Broken pipe" - https://phabricator.wikimedia.org/T78616#849414 (10Krinkle) 3NEW
[22:48:08] <Krinkle>	 YuviPanda: btw, looks like hashar filed a bug about the other broken instance: https://phabricator.wikimedia.org/T76250
[22:49:57] <YuviPanda>	 Krinkle: I commented.
[22:49:59] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Continuous-Integration: integration-slave1001.eqiad.wmflabs can't start, mount.nfs yields failure in name resolution - https://phabricator.wikimedia.org/T76250#849427 (10yuvipanda) So I looked at this today - our documented method for getting an interactive console for this seems...
[22:50:17] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Continuous-Integration: integration-slave1001.eqiad.wmflabs can't start, mount.nfs yields failure in name resolution - https://phabricator.wikimedia.org/T76250#849429 (10Krinkle) [[ https://wikitech.wikimedia.org/w/index.php?title=Release_Engineering/SAL&oldid=138053#December_2 |...
[22:51:34] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Labs-Team: Fix syslog error "nslcd[29117]: error writing to client: Broken pipe" - https://phabricator.wikimedia.org/T78616#849433 (10yuvipanda) Groups are:  50380 - tools 1003 - wmf 50062 - bastion 550 - svn  So apparently these groups are too large for nslcd to deal with. This d...
[22:51:47] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Labs-Team: Fix syslog error "nslcd[29117]: error writing to client: Broken pipe" - https://phabricator.wikimedia.org/T78616#849435 (10yuvipanda) p:5Triage>3Volunteer?
[23:50:58] <wikibugs>	 3Labs-Team: Make labs salt use instance names than ids - https://phabricator.wikimedia.org/T1154#849577 (10yuvipanda) So... talked to Ryan Lane some more and this is more complicated than expected ;)  Problem with `fqdn` is that deleting and creating an instance will cause complications because the salt (and pup...