[00:09:14] (03CR) 10Jforrester: [C: 032] #wikimedia-dev: No longer report integration/* [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/156269 (owner: 10Krinkle) [00:09:17] (03Merged) 10jenkins-bot: #wikimedia-dev: No longer report integration/* [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/156269 (owner: 10Krinkle) [00:11:33] (03CR) 10Jforrester: "Deployed and restarted." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/156269 (owner: 10Krinkle) [00:28:51] Coren: the scap job for beta has been failing a few times a day with errors like "Could not resolve hostname deployment-bastion.eqiad.wmflabs". Any known dns issues inside labs or just random flakiness? [00:32:10] bd808: could be just pdns failing again, open tickets to switch to gdns as in production and replace the middleware (mediawiki) with moniker [00:32:51] but would expect more error reports then.. [00:33:02] https://bugzilla.wikimedia.org/buglist.cgi?quicksearch=labs%20DNS&list_id=339684 [00:33:17] mutante: I get multiple reports a day of diamond failing because it can't resolve the hostname (of the machine it is on, noless) [00:33:26] but it always gets sorted so I don't really file bugs... [00:33:37] mutante: It seems to come and go. Probably temporary load. [00:33:53] bd808: yea, doesnt sound that new [00:34:58] * YuviPanda|zzz goes for realz [00:36:19] Labs dns is backed by ldap? There's your problem right there. ;) [00:41:04] right [00:41:23] also see https://bugzilla.wikimedia.org/show_bug.cgi?id=46818 [00:41:44] "broken in lots of ways" from Ryan , yay :) [00:43:10] maye it's not actually pri low ? [00:43:16] prio [00:43:53] YuviPanda|zzz: you should comment on that next time you see it, bye, laters [01:52:09] 3Wikimedia Labs / 3tools: Unable to explain queries on replicated databases - 10https://bugzilla.wikimedia.org/48875#c19 (10Tim Landscheidt) 5RESO/WON>3RESO/FIX My reading is that the essence of this bug, that is explaining queries on replicated databases, is now available. [02:01:41] 3Wikimedia Labs / 3tools: jstart doesn't signal out-of-memory kills to the user - 10https://bugzilla.wikimedia.org/50053#c2 (10Tim Landscheidt) My suggestion would send one (1) message if a job terminates that the user expects to be running continuously. [02:13:56] 3Wikimedia Labs / 3Infrastructure: Internal DNS look-ups fail every once in a while - 10https://bugzilla.wikimedia.org/70076 (10Tim Landscheidt) 3NEW p:3Unprio s:3normal a:3None DNS look-ups from Labs instances for the IP addresses of Labs instances fail every once in a while (all times UTC): | tool... [02:34:12] 3Wikimedia Labs / 3Infrastructure: Internal DNS look-ups fail every once in a while - 10https://bugzilla.wikimedia.org/70076#c1 (10Bryan Davis) This happens several times each day during the beta-scap-eqiad Jenkins job [0] as well: ssh: Could not resolve hostname deployment-mediawiki02.eqiad.wmflabs: Name o... [03:31:10] 3Wikimedia Labs / 3tools: Define rules for using Ganglia for individual tool statistics - 10https://bugzilla.wikimedia.org/48737#c3 (10Tim Landscheidt) Note that this *was* (and is) technically possible at http://ganglia.wmflabs.org/ when that was online; I think StatsD has no notion of authentication (yet).... [07:52:16] hello [09:43:41] 3Wikimedia Labs / 3Infrastructure: DNS can resolve to IPv6 address despite lack of IPv6 connectivity - 10https://bugzilla.wikimedia.org/70084 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None On deployment-bastion.eqiad.wmflabs we have a job pulling from Gerrit. It errored out with: INFO... [14:34:45] 3Wikimedia Labs / 3Infrastructure: DNS can resolve to IPv6 address despite lack of IPv6 connectivity - 10https://bugzilla.wikimedia.org/70084#c1 (10Marc A. Pelletier) That's actually behaviour mandated by the standards. DNS servers must return both A and AAAA RRs regardless of which protocol you reached it... [14:41:28] 3Wikimedia Labs / 3tools: Provide namespace IDs and names in the databases similar to toolserver.namespace - 10https://bugzilla.wikimedia.org/48625#c44 (10Marc A. Pelletier) Wait, if you want an additional DB named toolserver, where do you want me to put s51073__toolserver? [14:42:10] !log deployment-prep Root dirve full on deployment-mediawiki02; hhvm core files are the culprit [14:42:13] 3Wikimedia Labs / 3tools: Provide namespace IDs and names in the databases similar to toolserver.namespace - 10https://bugzilla.wikimedia.org/48625#c45 (10Marc A. Pelletier) (Also, if you want everyone to be able to select in a database, it normally needs to be named ending in _p) [14:42:14] Logged the message, Master [14:49:06] !log deployment-prep Moved hhvm core dumps to /data/project/hhvm-cores [14:49:09] Logged the message, Master [14:59:21] !log deployment-prep Resolved puppet git merge conflict on deployment-salt [14:59:23] Logged the message, Master [14:59:45] 3Wikimedia Labs / 3tools: Set up SPF/DKIM for tools.wmflabs.org - 10https://bugzilla.wikimedia.org/53101#c5 (10Marc A. Pelletier) The mail issue proper has long been fixed (that is, tools.wmflabs.org is the proper, canonical email domain for tool labs) but right now there is a technical limitation with our D... [15:03:58] 3Wikimedia Labs / 3tools: Package qdisplay and qtop - 10https://bugzilla.wikimedia.org/52437#c1 (10Marc A. Pelletier) 5NEW>3UNCO The fact that they /weren't/ packaged or puppetized is why they have not moved to eqiad during the migration -- and I don't recall anyone missing them. Are those programs stil... [15:20:19] 3Wikimedia Labs / 3tools: jstart doesn't signal out-of-memory kills to the user - 10https://bugzilla.wikimedia.org/50053#c3 (10Marc A. Pelletier) Wouldn't the default "Hey, I had to restart your job" from bigbrother fill that function? [15:21:53] Is SSH slow for anyone else? [15:23:31] Dispenser: I've had some strange hiccups [15:36:43] 3Wikimedia Labs / 3Infrastructure: Internal DNS look-ups fail every once in a while - 10https://bugzilla.wikimedia.org/70076#c2 (10Tim Landscheidt) Today the frequency of those occurences has increased quite a bit at Tools: | tools-webproxy.eqiad.wmflabs : Aug 27 00:03:35 : diamond : unable to resolve host... [15:39:20] typing echo seems to take a few minutes :-( [15:42:09] 3Wikimedia Labs / 3tools: Make Flow database available / accessible on Labs/Tools - 10https://bugzilla.wikimedia.org/67397 (10Tim Landscheidt) a:3Marc A. Pelletier [15:43:55] bd808: Dispenser yeah, am having network issues to labs as well [15:44:02] * YuviPanda pokes Coren [15:44:34] and this is sshing to a different labs instance [15:45:26] Database access seems to be normal [15:48:12] YuviPanda: I believe Coren is tinkering with DNS. Are you having connectivity issues or just DNS problems? [15:48:48] andrewbogott: unsure. connectivity issues, I think [15:49:04] my ssh just didn't go through, closed with: [15:49:08] Connection closed by 208.80.155.129 [15:49:09] ssh_exchange_identification: Connection closed by remote host [15:50:41] And that's still happening? Or was it a one-off? [15:50:58] andrewbogott: came through now, is still a bit slow tho [15:51:02] (other things are fast) [15:51:05] My traceroute to bastion-eqiad.wmflabs.org looks like a horrible road trip: boise > seattle > denver > chicago > dc > ashburn [15:51:46] Also my irc client is obviously having buffering issues, so maybe network issues at my provider [15:56:03] 3Wikimedia Labs / 3Infrastructure: DNS can resolve to IPv6 address despite lack of IPv6 connectivity - 10https://bugzilla.wikimedia.org/70084#c2 (10Antoine "hashar" Musso) Of all the build history I have, it only occurred twice for that job. So must be some very weird/rare issue. On deployment-bastion.eqia... [16:10:28] 3Wikimedia Labs / 3Infrastructure: DNS can resolve to IPv6 address despite lack of IPv6 connectivity - 10https://bugzilla.wikimedia.org/70084#c3 (10Marc A. Pelletier) (In reply to Antoine "hashar" Musso from comment #2) > Maybe if the v4 address is unreachable, git report the first failure message > instead... [16:16:14] is something up with bastion? my connection to it is super slow, but not to bastion2 [16:17:53] gifti: Oh, so the slowness isn't to all labs instances, just that one? [16:18:06] yes [16:18:21] * andrewbogott looks [16:21:09] gifti: I've learned nothing :( The instance itself seems fine [16:21:22] hm [16:22:06] !log deployment-prep killing `apt-get update` process running on deployment-bastion since Jun13 [16:22:07] it's ok again [16:22:09] Logged the message, Master [16:23:06] gifti: my ping times to the two boxes are pretty similar. [16:23:18] let me know if it slows down again, you're not the first person to mention things like this [16:23:25] ok [16:51:03] 3Wikimedia Labs / 3tools: Set up SPF/DKIM for tools.wmflabs.org - 10https://bugzilla.wikimedia.org/53101#c6 (10Marc A. Pelletier) 5ASSI>3RESO/FIX Turns out that adding the necessary attributes to the LDAP schema was mercifully straightforward. tools.wmflabs.org now has a proper SPF record. [17:00:43] 3Wikimedia Labs / 3Infrastructure: Internal DNS look-ups fail every once in a while - 10https://bugzilla.wikimedia.org/70076#c3 (10Marc A. Pelletier) 5NEW>3ASSI dnsmasq is a [bleep]ing piece of unreliable [bleep] that crumbles under the lightest load. I've been meaning to have a real DNS server in labs... [17:19:00] 3Wikimedia Labs / 3tools: Setup an icinga instance to monitor tools on tool-labs - 10https://bugzilla.wikimedia.org/51434#c6 (10Marc A. Pelletier) a:5Marc A. Pelletier>3Yuvi Panda Handing off to Yuvi, who is the gatekeeper of labmon1001 [17:20:45] 3Wikimedia Labs / 3tools: Package qdisplay and qtop - 10https://bugzilla.wikimedia.org/52437#c2 (10Tim Landscheidt) 5UNCO>3RESO/WON I never used them. [17:25:01] Coren: re: 70076, powerdns recursor would be a good option :) [17:27:13] 3Wikimedia Labs / 3tools: jstart doesn't signal out-of-memory kills to the user - 10https://bugzilla.wikimedia.org/50053#c4 (10Tim Landscheidt) That requires the job being managed by bigbrother. I just wanted to point out that IMHO one message per interaction is not spam, especially if it conveys important... [17:28:01] bblack: Probably the simplest one given that we use powerdns internally. But I remember Faidon saying something about wanting to look into using Unbound instead. [17:30:48] Coren: it might be interesting to look at. I donno, nlnetlabs has left a bad taste in my mouth recently. And pdns recursor is pretty awesome. [17:36:04] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c5 (10Aude) now we are getting Fatal error: Argument 1 passed to Wikibase\\EntityHandler::getTitleForId() must be an instance of Wikibase\\EntityId, Wikibase\\Dat... [17:47:59] bblack: But yeah; dnsmasq sucks slugs through a paper straw (sucks hard, doesn't get much done, and even when it works it's horrible) but we're stuck with it. Having a real DNS server the instances can hit would be ++good. [17:48:25] we're stuck with it because openstack has hooks into it, right? [17:48:46] Basically. [17:49:07] what does openstack actually do with it? send it ephemeral data on instance hostname/ip stuff? [17:49:33] oh it uses it for dhcp too [17:50:52] so, yeah, we could run e.g. pdns-recursor on some other secondary IP on labnet, and rework the betanatfix-via-dns stuff to populate it instead of dnsmasq, and move instance resolv.conf to point at that instead of dnsmasq, basically [17:51:02] and still leave dnsmasq running for dhcp or whatever that nova wants it for [17:54:44] multichill: re [17:55:01] OrenBochman: re what? P [17:55:27] What's up Oren? :-) [17:58:43] multichill: re [17:58:44] bblack: Possibly more robust would be to have pdns-recursor simply point to the dnsmasq as authoritative and rely on caching. [17:59:16] well yeah but then eventually things will break if dnsmasq breaks [17:59:29] the breakage will just be delayed to TTL expiries [18:00:17] OrenBochman1: Having connection problems? [18:00:21] dnsmasq breaks under load; and never for long. I should expect that having the recursor is going to releive the cause and hide the effects. [18:00:27] yes [18:00:55] I'm at Google Campus TLV - go figure [18:01:41] I was told by Yan that I should help out with WLM this year [18:02:38] so if any help is needed - perhaps you can add us to the WLM instance ? [18:02:52] Username? [18:03:02] me - Oren [18:03:06] rather oren [18:03:44] 3Wikimedia Labs / 3Infrastructure: Disable recordfail on labs instances - 10https://bugzilla.wikimedia.org/53159#c2 (10Marc A. Pelletier) 5NEW>3RESO/FIX The new images do not share that behaviour (both the newly-baked Precise and Trusty). [18:05:00] !log tools.heritage Added Oren to the project [18:05:03] Logged the message, Master [18:06:19] OrenBochman1: Please log any changes you do like I just did. Everything is in git, please commit your stuff [18:07:02] Plenty of bugs to fix in case you are bored or feel like being helpful [18:08:03] thanks! [18:08:28] 3Wikimedia Labs / 3Infrastructure: DNS can resolve to IPv6 address despite lack of IPv6 connectivity - 10https://bugzilla.wikimedia.org/70084#c4 (10Antoine "hashar" Musso) > Was the last occurrence of that bug recent? 9 hours ago: August 27th 9:00 UTC. I will have a look at git source code and maybe attem... [18:08:55] I'll be in touch before I make any changes [18:23:13] 3Wikimedia Labs / 3tools: 'crontab l' (with no dash) removes current crontab - 10https://bugzilla.wikimedia.org/69355 (10Marc A. Pelletier) 5PATC>3RESO/FIX [18:30:59] 3Wikimedia Labs / 3Infrastructure: DNS can resolve to IPv6 address despite lack of IPv6 connectivity - 10https://bugzilla.wikimedia.org/70084#c5 (10Antoine "hashar" Musso) 5NEW>3RESO/WON That might be some issue in curl which git relies upon. I am assuming the issue is with Gerrit being unreachable some... [18:56:15] 3Wikimedia Labs / 3wikitech-interface: Hostnames assigned to floating IP persist when deallocated - 10https://bugzilla.wikimedia.org/53816 (10Marc A. Pelletier) a:5Ryan Lane>3Andrew Bogott [18:57:32] 3Wikimedia Labs / 3wikitech-interface: Hostnames assigned to floating IP persist when deallocated - 10https://bugzilla.wikimedia.org/53816#c1 (10Andrew Bogott) Doesn't the web interface prevent release of an IP if it has hostnames? [19:03:39] Hi andrewbogott, yt? [19:03:53] I heard I should ping you about wmf/1.24wmfX and SemanticMediawiki [19:03:54] What's up? [19:04:20] I'm trying to upgrade webplatform.org (server mess) and it seems that SMW 2.0 is not ready at all [19:04:50] I thought to use mediawiki-vagrant and add in my own class a call such as `mediawiki::extension { 'SemanticMediaWiki': branch => '1.8.0.5', }` [19:04:58] am I doing something wrong? [19:05:29] I don't know offhand what the proper vagrant syntax is. but 1.8.0.5 seems like the right version to use [19:05:35] Well, at least, it's what wikitech uses. [19:05:52] What happens with that change? [19:07:15] its ok if you dont use vagrant. I heard the MediaWiki-Vagrant uses puppet though. [19:07:33] And one of the puppet manifest is about installing extensions, the one I gave. [19:08:14] So, when you install, do you use puppet, or you get a zip file and extract it like there’s no source control :/ [19:09:05] I don't quite understand your question. Are you asking how this would be done with vagrant, or how it's done on wikitech, or how you should do it? [19:09:09] Might be three different answers... [19:10:18] I'm asking how its done in wikitech [19:10:50] Wikitech handles extensions as git submodules. [19:11:00] As do other wmf wikis. [19:11:16] So, we have a particular checkout of SMW pegged to a particular MW branch [19:11:32] oh, submodules, right. [19:11:49] So if you have a local checkout of mediawiki version 1.24wmf15 [19:11:49] So, once I have my wmf clone, I make sure submodules are updated [19:11:59] you ought to be able to just submodule init SMW and related extensions [19:12:01] and then submodule update [19:13:16] Working with git submodules is so confusing. [19:13:32] Do you have SMW version declared as submodules in wmf branches? [19:13:55] i'm on 1.24wmv16 ATM [19:14:41] "we have a particular checkout of SMW pegged to a particular MW branch" I think that's the same as 'declared as submodules in wmf branches' [19:14:58] yes, that answers my question. [19:15:06] I would like to set up a web-accessible test wiki for a new extension intended for deployment on Wikipedia. Is there a good instruction page on how to do that? I have admin access to relevant labs project, but I'm lost when it comes to setting an instance up for this purpose. [19:16:18] ragesoss: https://wikitech.wikimedia.org/wiki/Labs-vagrant [19:16:19] :) [19:16:22] ragesoss: are you comfortable with vagrant? It's pretty simple to turn a vagrant config into a publicly-reachable labs box [19:16:49] andrewbogott: I wouldn't say 'comfortable', but I've certainly used it a lot. [19:17:03] yeah, so, then, as Yuvi says... [19:17:07] :) [19:17:12] cool, thanks. [19:17:32] Make sure you set up a security group with web access for your instance! [19:20:33] (03PS1) 10coren: Add .jsubrc support to jsub/jstart [labs/toollabs] - 10https://gerrit.wikimedia.org/r/156608 (https://bugzilla.wikimedia.org/54054) [19:21:51] oh, i didnt know there were two Vagrant projects [19:22:18] labs-vagrant is just using the mw-vagrant code in a labs instance [19:22:31] same puppet, different way of applying it to the node [19:22:36] oh, so your local VM controls a remote one? [19:23:22] It's actually a labs puppet role that checks out mw-vagrant and a script to run the puppet code that provides [19:23:35] I found a site that uses MW 1.23 with SMW 1.9 [19:23:36] http://www.discoursedb.org/wiki/Special:Version [19:24:15] * renoirb But doesn’t mean it works, I do use SMW 2.0 w/ 1.24wmf, and it doesnt work well http://docs.webplatform.org/wiki/Special:Version [19:24:42] (03PS2) 10coren: Add .jsubrc support to jsub/jstart [labs/toollabs] - 10https://gerrit.wikimedia.org/r/156608 (https://bugzilla.wikimedia.org/54054) [19:25:42] Coren: -mem 1G ALL the things ;p [19:26:17] but yeah, that's a good thing to have. thanks. [19:26:43] It's a long-time Betacommand request, but it makes sense. [19:27:58] 3Wikimedia Labs / 3tools: symlinks not always followed - 10https://bugzilla.wikimedia.org/54056#c3 (10Marc A. Pelletier) 5NEW>3RESO/INV This may have been a temporary issue at the time. Unable to reproduce. [19:33:58] 3Wikimedia Labs / 3tools: Make exec hosts as submit hosts too - 10https://bugzilla.wikimedia.org/54786#c2 (10Marc A. Pelletier) 5NEW>3RESO/FIX Despite some hesitation about possible runaway jobs, this has been done. Note that the jsub/jstart tools are not available on the exec nodes by design; submittin... [19:41:22] bd808, do you know if mediawiki-vagrant actually do run git submodule update --init --recursive? [19:41:39] ... in the /vagrant/mediawiki ... or I have to do it manually? [19:42:00] 3Wikimedia Labs / 3tools: Impose per-user home directory quota in Tool Labs - 10https://bugzilla.wikimedia.org/55603#c1 (10Marc A. Pelletier) 5NEW>3RESO/WON The impositions of hard quota should be avoided unless we are really tight on resources or there are actual issues caused by overuse which is not cu... [19:42:43] 3Wikimedia Labs / 3tools: not able to ssh to tools-dev from tools-login - 10https://bugzilla.wikimedia.org/55945#c4 (10Marc A. Pelletier) 5NEW>3RESO/INV Unable to reproduce. [19:42:50] Could we get wine installed? [19:42:57] renoirb: It should have done it. Look at git::clone to see that this is the default behavior [19:43:11] renoirb: Running manually is non-destructive too [19:43:39] ok [19:43:41] Dispenser: you could file a bug, but I suspect 'no' [19:43:45] good, its building now [19:44:27] renoirb: huh I'm sort of wrong. git_submodule_update is only triggered if $ensure == 'latest' is set and we don't do that for the main mw clone [19:44:39] So looks like you have to do it yourself [19:45:01] Which kind of makes sense because we don't normally run a branch that uses submodules [19:45:48] It does add ``--recurse-submodules` to the clone command by default but I'm not sure what that does [19:46:16] That was my feeling. I still haven’t seen any submodule stuff happening in my last `vagrant up` rebuilds [19:47:00] renoirb: `vagrant git-update` may do what you want [19:47:43] 3Wikimedia Labs / 3tools: Provide incomplete (running) dump files in public datasets too - 10https://bugzilla.wikimedia.org/56092#c1 (10Marc A. Pelletier) 5NEW>3RESO/WON Tool labs has access to exactly what is available from the source. [19:49:59] 3Wikimedia Labs / 3tools: User accounts should have a replica.my.cnf - 10https://bugzilla.wikimedia.org/57485#c5 (10Marc A. Pelletier) 5NEW>3RESO/INV Not actionable (no examples given) [19:50:03] 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708 (10John F. Lewis) p:5Unprio>3Normal [19:50:15] 3Wikimedia Labs / 3tools: User accounts should have a replica.my.cnf - 10https://bugzilla.wikimedia.org/57485#c6 (10Yuvi Panda) Yup, I was talking about user accounts on toollabs, and they seem to have one. I iz blind? [19:59:19] oh, thx bd808, will do that ! [19:59:34] Gosh the vagrant build with puppet is long :/ [20:00:03] 3Wikimedia Labs / 3tools: Can't create functions on p50380g50790__wanderwiki database - 10https://bugzilla.wikimedia.org/57558#c6 (10Marc A. Pelletier) 5ASSI>3RESO/FIX This should be fixed now. [20:00:06] Dispenser: Surely, thou jests? [20:00:20] No [20:00:35] Dispenser: What possible open source windows-only program could you want to run in Labs? [20:00:42] (That isn't .net) [20:00:49] (for which we have Mono) [20:01:06] As far as I know, wine requires X to start up. I don't think there's a console-only version (even though it can run console-only apps) [20:01:41] Its not open source, but I'd like to try using it with my data [20:01:49] andrewbogott: IIRC, it's *possible* to build wine with no X support, but the distros don't do that of course. [20:02:01] Dispenser: If it's not open source, then it's not allowable on Labs in the first place. [20:02:06] Aren't there null X servers? [20:02:27] Although with wine we could install cygwin and run openstack on cygwin thus adding another layer to the onion... [20:02:49] * Coren throws rotten tomatoes at andrewbogott. [20:03:20] Dispenser: I don't think you're allowed to put non open source code on labs, and I think you of all people should know [20:03:40] andrewbogott: Though I doubt Openstack actually *builds* on cygwin. I doubt there is a kvm backend for it. :-) [20:04:01] hm, probably not :( [20:04:13] Coren: heh, openstack with CreateProcessEx backend? :) [20:04:14] I'm actually testing Pixelizer via PuTTY+Xming [20:04:40] Dispenser: I suggest you test those things on a local VM of yours. :-) [20:04:42] andrewbogott: Coren http://docs.openstack.org/trunk/config-reference/content/hyper-v-virtualization-platform.html though :) [20:06:32] Comments like those are what has me believing the staff isn't eating their own dog food [20:07:25] Dispenser: 'open source software only on labs' - how does that relate to 'not eating dog food'? [20:09:06] Dispenser: You are welcome to point out exactly /what/ non-open-source software I or Labs uses. [20:09:13] YuviPanda: No the 'Access our stuff over our high latency link' [20:10:03] you're not making much sense at this point, Dispenser. [20:11:15] Dispenser: You're missing the point. We're not telling you "you should do this stuff on your laptop because the remoteness is fun" we're saying "you should do this stuff on your laptop because it's not open source and thus strictly forbidden on labs" [20:11:49] I'm testing image mosaic software. I've downloaded about 10 GB of thumbnails from Commons. Results and performance characteristics vary a lot between packages. [20:12:46] Dispenser: No doubt the metrics of proprietary software may be interesting as comparison points but moot since they would never be allowed anywhere on the WMF infrastructure. [20:14:03] Coren: Could I put in the part of Labs that under Toolserver legacy proprietary software? [20:14:38] Is the software legacy? [20:15:02] Dispenser: I don't think there's any proprietary software from toolserver that survived the migration [20:15:39] Dispenser: There is no such part. You may *ask* Legal to okay a project where you could run non-open software; but I wouldn't hold my breath unless the software is (a) irreplacable, (b) you have a suitable license that allows you to run it on the project and (c) you make a compelling case why we should host it. [20:16:01] Dispenser: For reference, no such okay has ever been given. [20:16:12] YuviPanda: /home/dispenser/absurdsize/bin/pngout [20:16:43] Dispenser: err, if that's proprietary software and you're using it on toollabs, I highly suggest you remove it. [20:17:09] It's Toolserver legacy and required for generating the report [20:17:12] Dispenser: Let me clarify Yuvi's statement. This isn't a suggestion; you are violating the TOU. [20:18:05] Silke_WMDE: ^ [20:18:50] Dispenser: pngout is a png compressor. How is it 'required'? [20:19:00] there's also http://pmt.sourceforge.net/pngcrush/ and tons of other alternatives [20:19:23] (that are open source) [20:19:27] That didn't work when I tested it [20:19:38] or, you know, just don't care about the 10% size difference in the png? [20:47:58] I tried advpng and optipng and anything else I could get compiled on the Toolserver. IIRC this was the only compressor that worked well on the warez suspects. [20:49:26] Dispenser: 'warez suspects'? [20:49:36] https://commons.wikimedia.org/wiki/User:Dispenser/Absurd_overhead [20:51:17] warez implies illegal, can you please confirm if you rightfully acquired the software in question, Dispenser ? [20:51:50] (not that the distinction would make a difference for the use of such software on toollabs) [20:51:56] greg-g: you're missing the point [20:52:02] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29/Archive_117#File_Format_Expert [20:52:06] greg-g: it's about abusing images to store data, presumably warez [20:52:20] ahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh [20:52:23] my apologies [20:52:24] :) [20:52:25] We found password protected 7z and RAR files [20:53:03] * greg-g moves on [20:53:08] steganography on commons? interesting, i was wondering about that [20:53:19] Dispenser: I mean, it's clear that your work is valuable, but wouldn't a simple -> bmp -> back to png be good enough? [20:53:32] Maybe that's why there are 300-MB SVGs everywhere [20:53:37] Dispenser: it doesn't have to be small, it just has to be 'the same but without the crap we don't want' [20:53:48] Or was it kb. [20:53:53] Still, way too big [20:54:53] Can the lightty config be configured so that if a file doesn't exist in public_html, it will check in cgi-bin? [20:54:57] <^d> !log integration upgraded elasticsearch to 1.3.2 on integration-slave100[1-3] [20:55:00] Logged the message, Master [20:55:06] That's basically column 1, column 3 is general compression (somewhere on commons is floating a 300 MB TIFF that compress to 24 KB), column 2 is image optimization (discarding excessive metadata, e.g. thumbnail) [20:59:31] 3Wikimedia Labs / 3deployment-prep (beta): Rename labswiki to deploymentwiki - 10https://bugzilla.wikimedia.org/70108 (10Andrew Bogott) 3NEW p:3Unprio s:3normal a:3None I'm in the middle of some work to get wikitech to function as part of the deployment system: https://gerrit.wikimedia.org/r/#/c/155... [21:01:13] I'm getting the feeling WMF rather warez hidden away than allow a toolchain that has one piece of proprietary software to check it [21:08:02] Dispenser: are you missing a verb? i can't figure out what "rather warez hidden away" means [21:08:26] jackmcbarn: http://www.wikihow.com/Store-a-Rar-or-Zip-in-an-Image-File [21:08:56] Dispenser: i know what hiding warez means, i just can't make sense of it in that place in that sentence [21:09:08] rather have warez hidden away* [21:09:14] ah. [21:11:05] Dispenser: I don't understand why you would need proprietary software for this: isn't dump to bitmap & re-encode good enough? [21:11:17] because, metadata [21:13:24] Dispenser: oh, right. So it's not as simple as I thought. Still, why would you need proprietary software for it? [21:13:57] Because it was the only package that worked out of those I tested and compiled [21:14:05] Is is possible that a rescheduled job has left a ghost of the old program still running? [21:14:43] Dispenser: as in: it's the only program that gave you a new png that still had the relevant metadata but not the other crap that was stored? [21:16:00] pretty much and didn't increase the filesize [21:17:19] Dispenser: but we don't care about the filesize, we just care about not storing other people's crap.... [21:17:40] Coren: Could there be a ghost in the machine? [21:17:43] valhallasw`cloud: https://commons.wikimedia.org/wiki/Category:Fireworks_PNG [21:18:10] a930913: check qstat, but if it's rescheduled, it should just be restarted at another hots automatically [21:18:48] valhallasw`cloud: Yeah, but it seems to be running multiple times :/ [21:18:57] a930913: weird. Check qstat... [21:19:18] Should we delete files in that category for being proprietary [21:19:29] 3Wikimedia Labs / 3deployment-prep (beta): Rename labswiki to deploymentwiki - 10https://bugzilla.wikimedia.org/70108#c3 (10Andrew Bogott) Renaming requires a manual rename of the database (probably via a dump and recreation) and the attached patch. And... anything else? [21:19:50] valhallasw`cloud: I have, 192887 0.71469 vandalstat tools.cluest Rr 08/27/2014 12:33:53 continuous@tools-exec-02.eqiad 1 [21:19:59] Dispenser: deleting: no. unpacking and recompressing: maybe [21:20:26] Dispenser: if the pngs become bigger in doing that: *shrug* [21:20:42] a930913: that's the only entry? then at least there's not multiple jobs running [21:24:36] valhallasw`cloud: So why is my output file being updated multiple times? :/ [21:24:52] a930913: define 'being updated multiple times' [21:25:07] a930913: if you run multiple times, the output file is concatenated. Is that what you mean? [21:25:44] valhallasw`cloud: The program appends a line to a file every minute. [21:26:05] valhallasw`cloud: The line is being added multiple times. [21:26:11] a930913: weeeiiirrrrd [21:26:29] 3Wikimedia Labs / 3deployment-prep (beta): VisualEditor: All kind of search is extremely slow in Betalabs which includes searching for matching link target names, images, category names, templates etc - 10https://bugzilla.wikimedia.org/70103 (10Greg Grossmeier) p:5Unprio>3Highes a:3None [21:27:52] this labs-vagrant thing is great! I'm never running my own local vagrant again! [21:28:03] ragesoss: \o/ [21:28:25] valhallasw`cloud: WhatdoIdo? :s [21:28:37] DO YOU HEAR THAT, RANDOM ERRORS ON MY LOCAL MACHINE? THIS MEANS YOU. [21:28:45] a930913: I'm trying to check the online process list, but it's sloooooow [21:28:52] a930913: I'd say: jstop ALL the things [21:28:56] and retry from there [21:29:02] valhallasw`cloud: Ah yes, mine just loaded. [21:29:03] ragesoss: mediawiki-vagrant works well for local dev too [21:29:19] valhallasw`cloud: Only records one instance of the job. [21:29:20] bd808: for some definition of 'works' :-p [21:29:28] 3Wikimedia Labs / 3deployment-prep (beta): All kind of search is extremely slow in Betalabs which includes searching for matching link target names,images, category names, templates etc - 10https://bugzilla.wikimedia.org/70103 (10Greg Grossmeier) [21:29:41] valhallasw`cloud: them's fightin words ;) [21:30:01] bd808: that's what they say, except my machine is deciding today that it can't boot up a working instance. [21:30:05] It's a lot better than building a wiki stack by hand [21:30:18] yeah, I've been using mediawiki-vagrant. [21:30:46] but who knows what I updated on my debian system (and when) that is now causing problems. [21:30:46] My next big trick for labs-vagrant is to make it actually use Vagrant and docker [21:30:57] systemd had something to do with it, but I think I'm past that part. [21:31:02] a930913: so did that stop the writing to the file? [21:31:13] 3Wikimedia Labs / 3deployment-prep (beta): All kind of search is extremely slow in Betalabs which includes searching for matching link target names, images, category names, templates etc - 10https://bugzilla.wikimedia.org/70103#c1 (10Nik Everett) I see it taking a long time too. The load on the search serve... [21:31:31] valhallasw`cloud: But I'll lose my low job number :( [21:31:42] a930913: LOL [21:34:35] Ok, job number lost. [21:35:48] 3Wikimedia Labs / 3wikitech-interface: Hostnames assigned to floating IP persist when deallocated - 10https://bugzilla.wikimedia.org/53816#c2 (10Marc A. Pelletier) I'm uncertain; I ran into this I think after a deleted /project/ ended up with IPs back in the pool. [21:36:24] Ok, seems to have stopped. [21:36:33] * valhallasw`cloud consoles a930913 [21:36:45] a930913: but maybe you can try to get number 930913 at some point! :-D [21:36:59] oh, wait, we're in the millions already [21:37:41] Your job 3507314 ("vandalstats") has been submitted [21:39:22] valhallasw`cloud: At first glance it seems to have been fixed. [21:39:30] a930913: weeiiirrddd [21:40:18] 3Wikimedia Labs / 3deployment-prep (beta): hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - 10https://bugzilla.wikimedia.org/69979#c5 (10Antoine "hashar" Musso) I have deleted the 2GB+ core files on mediawiki02:/tmp/ [21:41:54] valhallasw`cloud: Do you want to file a bug? [21:42:20] a930913: *shrug* don;t get what caused it, so I'm not even sure what to report [21:42:44] YuviPanda: What's the current log-rotation of proxy-logs? daily or hourly? [21:43:10] * hedonil needz the data and wants bug 59222 to be closed soon [21:43:18] hedonil: daily I think [21:44:26] YuviPanda: can you take a second look and provide, let's say 3 complete logiles (sanitized) in /shared/sample-nginx-log/ ? [21:44:40] f [21:44:55] hedonil: I already put a sanitized logfile somewhere, no? and mentioned it in the bug... [21:45:35] YuviPanda: yeah you put a sample in that place, but I needz more data to test the .py routine [21:46:30] hedonil: ah, hmm. I've to go off now, sadly :( I'll get them to you tomorrow? [21:46:36] hedonil: the format is exactly the same, tho [21:47:17] YuviPanda: the point is, I need to know when hours /days cahnge over logfiles - that's why [21:48:23] hedonil: ah, hmm. that we can kinda fudge that to what we want. [21:48:25] YuviPanda: tomorrow is also ok, put it in /shared/sample-nginx-log/ , I'll pick it up from this location [21:48:32] cool, will do [22:03:03] 3Wikimedia Labs / 3deployment-prep (beta): hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - 10https://bugzilla.wikimedia.org/69979 (10Greg Grossmeier) p:5Unprio>3Highes [22:03:43] 3Wikimedia Labs / 3deployment-prep (beta): hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - 10https://bugzilla.wikimedia.org/69979 (10Greg Grossmeier) [22:04:13] 3Wikimedia Labs / 3Infrastructure: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - 10https://bugzilla.wikimedia.org/69601 (10Greg Grossmeier) [22:04:55] Coren: Unable to run job: unable to send message to qmaster using port 6444 on host "tools-master.eqiad.wmflabs": can't resolve host name. [22:05:05] from cron@tools-submit [22:05:14] 3Wikimedia Labs / 3deployment-prep (beta): hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - 10https://bugzilla.wikimedia.org/69979#c6 (10Greg Grossmeier) a:3Bryan Davis Since Bryan and Giuseppe were working on this issue this morning, I'm assigning to Bryan. :) [22:05:18] dnsmasq giving us lots of "love" again. [22:06:57] 3Wikimedia Labs / 3tools: jsub and utf8 - 10https://bugzilla.wikimedia.org/58784#c10 (10Marc A. Pelletier) 5NEW>3RESO/WOR Left without comment for >six months; reopen if the issue is still relevant. [22:07:35] Coren: looking for a resilient dns server? "Dnsmasq provides network infrastructure for small networks... suitable for resource constrained routers and firewalls. It has also been widely used for tethering on smartphones and portable hotspots.." [22:07:37] lol [22:07:58] 3Wikimedia Labs / 3tools: -once is not checked correctly by jstart randomly - 10https://bugzilla.wikimedia.org/58145#c2 (10Marc A. Pelletier) Has this issue occurred again in the past six months or so? [22:08:02] Yeah, we *know* dnsmasq sucks vastly. [22:08:17] Coren: SCNR [22:08:40] <\3 dnsmasq [22:08:56] Yeah, I'm putting "put a real DNS server for labs" atop my list. [22:09:58] 3Wikimedia Labs / 3deployment-prep (beta): Search and page loads extremely slow on beta cluster (cause being investigated) - 10https://bugzilla.wikimedia.org/70103#c2 (10Greg Grossmeier) Not this slow. [22:10:35] !log deployment-prep restarted varnish on deployment-cache-text02 [22:10:37] Logged the message, Master [22:10:53] The problem is actually simple: dnsmasq crumbles under the lightest load. I actually fear looking at the source, I expect to find a for(;;) loop that does a blocking recvmsg(), processes stuff, then sends a reply. :-) [22:19:14] !log deployment-prep restarted varnish (again) on deployment-cache-text02 [22:19:17] Logged the message, Master [22:26:42] !log deployment-prep when restarting varnish on deployment-cache-text02, don't forget that there are 2 varnish services (varnish and varnish-frontend) [22:26:45] Logged the message, Master [22:30:28] 3Wikimedia Labs / 3deployment-prep (beta): Search and page loads extremely slow on beta cluster (cause being investigated) - 10https://bugzilla.wikimedia.org/70103#c3 (10Greg Grossmeier) 18:29 < bd808> For some currently unknown reason varnish is not caching anything [22:34:30] Can url.rewrite-once be made case insensitive? [22:37:07] !log deployment-prep restarting udp2log-mw on deployment-bastion. It keeps crashing since fiarly recently [22:37:09] Logged the message, Master [22:39:13] 3Wikimedia Labs: Request to access redacted webproxy logfiles of (Tool) Labs - 10https://bugzilla.wikimedia.org/59222#c17 (10metatron) 5UNCO>3NEW Working on the little routine right now. It will provide 2 formats. - query-strings are stripped from both: request and referer 1) "Tools Aggregate Format" Pe... [22:43:28] 3Wikimedia Labs / 3tools: Performance problem on database server s5 using commonswiki - 10https://bugzilla.wikimedia.org/67602#c8 (10Marc A. Pelletier) Any news on this that isn't otherwise tracked with Sean regarding performance issues with the new database? [22:44:04] 3Wikimedia Labs / 3deployment-prep (beta): hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - 10https://bugzilla.wikimedia.org/69979#c7 (10Bryan Davis) Bandaid solution: $ cat cleanup-hhvm-cores #!/usr/bin/env bash sudo mv /tmp/hhvm.*.core /data/project/hhvm-cores sudo mv /v... [22:46:57] !log deployment-prep blackholed an IP address on deployment-cache-text02 and deployment-cache-mobile03 , it was causing hundred of requests per seconds and overloaded the beta cluster. Use route -n to find the IP [22:46:59] Logged the message, Master [22:48:36] !log the IP is a known security audit [22:48:36] the is not a valid project. [22:48:46] !log deployment-prep the IP is a known security audit. See Chris Steipp. [22:48:48] Logged the message, Master [22:49:58] 3Wikimedia Labs / 3tools: Create status page for Tool Labs - 10https://bugzilla.wikimedia.org/58795#c4 (10Marc A. Pelletier) 5NEW>3RESO/WON This will be made obsolete by labmon. [22:50:28] 3Wikimedia Labs / 3tools: commonswiki_p.hashs view is broken on Wikimedia Labs - 10https://bugzilla.wikimedia.org/59684#c2 (10Marc A. Pelletier) 5ASSI>3RESO/FIX The view is gone. [22:50:28] 3Wikimedia Labs / 3tools: Audit security groups - 10https://bugzilla.wikimedia.org/60144#c1 (10Marc A. Pelletier) 5NEW>3RESO/INV Anything intended "prior to the move" is not all that relevant today. :-) [22:53:12] !log deployment-prep removed the blackhole ip route from deployment-cache-text02 and deployment-cache-mobile03 [22:53:14] Logged the message, Master [22:53:50] hashar: Was that the whole problem, just one person doing too many requests? Feel free to block it if it's causing problems. I'll communicate to him if needed. [22:56:12] Krinkle: Chances to get things merged to /static? and enable caching directives? I want to switch all static resources to /static asap [22:56:40] I don't know much about how it works. I'm mostly co-owner as backup. I don't maintain it. [22:56:50] Ask one of the primary maintainers. [22:57:08] I'm just here to complain and help someone else take over if the current maintainers leave. [22:57:37] Krinkle: ;-) right now, you're the only one at hand [22:57:43] 3Wikimedia Labs / 3deployment-prep (beta): Search and page loads extremely slow on beta cluster (cause being investigated) - 10https://bugzilla.wikimedia.org/70103#c4 (10Bryan Davis) That may have been my user-agent doing something strange. Antoine also found that we were undergoing a high rate vulnerabilit... [22:58:13] 3Wikimedia Labs / 3tools: [NewWeb] First few requests always return 404 after lighttpd restart - 10https://bugzilla.wikimedia.org/60201#c3 (10Marc A. Pelletier) 5NEW>3RESO/WOR Sorry, it's not really possible to do so (the script can know when the job has been submitted, not when the process is ready to a... [22:58:33] Krinkle: Look what I'm making :) https://tools.wmflabs.org/cluestuff/AmINeededToCounterVandal [23:00:26] Krinkle: fully automated cv, heh [23:00:48] bad news for all the vandal fighters out in the field [23:01:18] Hmm, should it be "Am I", or "Are you"? [23:03:39] !log deployment-prep Blacklisting the security audit IP again on deployment-cache bits01 mobile03 and text02 [23:03:42] Logged the message, Master [23:03:44] bd808: csteipp ^^^ [23:04:03] bd808: csteipp: will poke the security audit volunteer and leave it blackholed till the script is throttled [23:04:34] Perhaps they are testing DoS handling? [23:05:18] heh. if only we had DoS handling in beta other than waking up hashar [23:06:11] * hedonil is not in the vandal-fighter whatsoever business , but is sceptic anyways. It's like removing all disagreements from community. -> quiet but dead. [23:06:29] SOP for DoS: run round yelling 'Someone's going to get fired for this' [23:06:31] 3Wikimedia Labs / 3tools: Start portgranter and provide portgrabber on every exec node - 10https://bugzilla.wikimedia.org/60240#c4 (10Marc A. Pelletier) 5NEW>3RESO/FIX This is now possible and supported via the (increasingly misnamed) tools-webgrid-tomcat. See https://wikitech.wikimedia.org/wiki/Nova_Re... [23:06:43] 3Wikimedia Labs / 3deployment-prep (beta): Search and page loads extremely slow on beta cluster (cause being investigated) - 10https://bugzilla.wikimedia.org/70103#c5 (10Antoine "hashar" Musso) We have some security audit being run on the beta cluster. Unfortunately the script is not throttled and cause a fa... [23:07:33] 3Wikimedia Labs / 3deployment-prep (beta): hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - 10https://bugzilla.wikimedia.org/69979#c8 (10Bryan Davis) a:5Bryan Davis>3None Unlicking this cookie. The core (ha punny) problem remains but hopefully someone on the hhvm team ca... [23:07:44] hashar: Sounds good. Sorry to keep you up! [23:09:04] Damianz: o/ [23:10:33] 3Wikimedia Labs / 3tools: Install ogc on tools - 10https://bugzilla.wikimedia.org/60449#c1 (10Marc A. Pelletier) It's not clear that this is generally "system" installable; wouldn't it be easier in general to install it in the tool itself? [23:10:39] csteipp: would definitely need a good strategy for the security audit [23:10:54] csteipp: probably have some rule in varnish to direct such traffic on a dedicated app server [23:11:11] csteipp: this way the sec audit would not cause too much issues for beta other use cases [23:12:07] beta.clone('new project name') // :) [23:12:14] 3Wikimedia Labs / 3tools: Soften qdel behaviour from KILL - 10https://bugzilla.wikimedia.org/61102#c5 (10Marc A. Pelletier) 5NEW>3RESO/FIX The grid has been adjusted to use SIGTERM by default now; this problem should be solved. [23:12:32] 3Wikimedia Labs / 3deployment-prep (beta): hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - 10https://bugzilla.wikimedia.org/69979#c9 (10Greg Grossmeier) Resseting assignee and priority as this is now no longer an OMG! situation. For the record: gjg@deployment-bastion:/data... [23:12:43] 3Wikimedia Labs / 3deployment-prep (beta): hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - 10https://bugzilla.wikimedia.org/69979 (10Greg Grossmeier) p:5Highes>3Normal [23:15:32] A message that you sent could not be delivered to one or more of its [23:15:33] recipients. This is a permanent error. The following address(es) failed: [23:15:33] tfaprotbot.maintainers@tools.wmflabs.org [23:15:33] (generated from tools.tfaprotbot@tools.wmflabs.org) [23:15:33] Unrouteable address [23:15:47] ireas: that happened quickly :-) [23:15:48] Coren: do I need to do antyhing about ^? it's a crontab email [23:16:32] 3Wikimedia Labs / 3tools: running job with SGE doesn't get the right environment when using -once -continuous - 10https://bugzilla.wikimedia.org/65491#c3 (10Marc A. Pelletier) Actually, jsub passes -V in neither case; but the gridengine is really random about when it decides to start subshells or not. I str... [23:17:27] legoktm: Hmm. That's odd. [23:17:48] would you like me to pastebin the full email? [23:17:54] we can send cronspam to project owners now? nice!! [23:18:19] legoktm: That shouldn't be necessary. Lemme go take a peek at the logs. [23:18:29] It's not like ops ever complained when I started sending them thousands of emails.... not [23:19:07] ok [23:19:41] hedonil: I’ve already been in IRC ;) [23:20:16] * hedonil was kind of blind [23:21:04] hedonil: … but not in this channel *G* [23:21:56] Damianz: Why has CBNGRelay changed, and why is it now CBNGRelay9001? [23:22:39] legoktm: I see no reason why it failed beyond "unroutable address" and yet mail to maintainers of other tools go through fine. Can you try again? [23:23:04] Because there was like 15 duplicate processes of everything running on the grid... which is awesome considering it's suppose to enforce one. Killed all the jobs and restarted them, so it's back to 1 [23:24:03] Coren: well it was a cron email and it runs every hour...should I just wait 36 minutes? [23:24:35] That, or you can just try forcing some email to be sent. [23:25:14] how? [23:27:34] date | mail csbot.maintainers@tools.wmflabs.org [23:27:51] Well, that's to me. Substitute the right tool. :-) [23:33:25] I did that but I can't find the email, I probably have some crazy filter set up for it [23:33:34] tools.legobot@tools-login:~$ date | mail tfaprotbot.maintainers@tools.wmflabs.org [23:33:34] specifically [23:34:22] It may take a minute or two to get there, especially if you're gmail-bound. [23:34:54] oh there it is [23:34:58] email looks fine [23:39:45] 3Wikimedia Labs / 3deployment-prep (beta): udp2log logs are not properly rotated / daemon restarted - 10https://bugzilla.wikimedia.org/70112 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None We have udp2log-mw service on deployment-bastion which write to log files under /data/project/logs/... [23:50:58] 3Wikimedia Labs / 3deployment-prep (beta): udp2log logs are not properly rotated / daemon restarted - 10https://bugzilla.wikimedia.org/70112#c1 (10Antoine "hashar" Musso) /var/log/udp2log/udp2log.log may have some clue. Not we had a huge number of requests send for the last few days, so that might have prev... [23:51:28] 3Wikimedia Labs / 3deployment-prep (beta): Search and page loads extremely slow on beta cluster (cause being investigated) - 10https://bugzilla.wikimedia.org/70103#c6 (10Greg Grossmeier) gjg@deployment-bastion:/data/project/logs$ grep -c REDACTED xff.log 4960 gjg@deployment-bastion:/data/project/logs/archiv... [23:53:28] 3Wikimedia Labs / 3deployment-prep (beta): Search and page loads extremely slow on beta cluster (cause being investigated) - 10https://bugzilla.wikimedia.org/70103#c7 (10Antoine "hashar" Musso) The udp2log-mw service on deployment-bastion.eqiad.wmflabs logs the average number of packets it receives per secon... [23:54:19] Damianz: The feed also changed, some ANN scores aren't being output? [23:56:13] 3Wikimedia Labs / 3deployment-prep (beta): Search and page loads extremely slow on beta cluster (cause being investigated) - 10https://bugzilla.wikimedia.org/70103#c8 (10Greg Grossmeier) 5NEW>3RESO/FIX Closing this. Thanks Rummana for the heads up and for all who helped debug this multilayered issue.