[02:33:23] 3Wikimedia Labs / 3tools: Is it possible to install w3c-markup-validator? - 10https://bugzilla.wikimedia.org/69105#c3 (10Tim Landscheidt) Indeed the installation fires up an actual webserver. We could work around that and shut it down with Puppet, but if you could live with installing the tar balls locally... [05:45:12] Why some labels of xtools doesn't translated? we translated them in translatewiki 1 month ago! [07:45:40] 3Wikimedia Labs / 3tools: Is it possible to install w3c-markup-validator? - 10https://bugzilla.wikimedia.org/69105#c4 (10Rainer Rillke @commons.wikimedia) 5ASSI>3RESO/WON (In reply to Tim Landscheidt from comment #3) > if you could live with installing the tar balls locally Sure. Thanks for investigating. [07:55:50] !log deployment-prep https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ is broken :-/ [07:55:53] Logged the message, Master [07:57:22] !log deployment-prep fixing ownerships under /srv/scap-stage-dir/php-master/skins some files belong to root [07:57:24] Logged the message, Master [08:02:30] !log deployment-prep beta-code-update-eqiad is running again [08:02:32] Logged the message, Master [09:01:08] 3Tool Labs tools / 3[other]: Migrate to Tool Labs: https://toolserver.org/~vvv/adminstats.php - 10https://bugzilla.wikimedia.org/61030#c2 (10Glaisher) (In reply to Sjoerd de Bruin from comment #1) > Any update on this? It's a pretty useful tool, there is no alternative for > it. There is: [[toollabs:xtools/... [12:07:38] springle: hey! mind if I pick your brain for a moment? [12:15:21] YuviPanda: so long as tired, coffee-driven responses suit you ;) [12:19:03] springle: heh :) [12:19:27] springle: soo, assuming we wanted to replicate one of the tables from the eventlogging db (log), and have things cleared with legal, etc. how hard would it be? what would it take? [12:20:09] ugh [12:20:24] wait [12:20:33] replicate to labs? [12:20:42] springle: yes [12:20:52] springle: this is something the research team wants to do and I would like to help [12:21:02] purely public data (like Schema:PageMove) but much better structured [12:21:20] this ia the same research team that already has access to eventlogging upstream? [12:21:38] springle: yes, but for others outside :) Also wikimetrics uses labsdb. [12:21:56] springle: me and halfak would happily talk to legal, etc to get those things done. I'm trying to see how hard /easy it is technically [12:22:28] my concern would be that eventlogging is a slippery slope with high write load. how sure are we it would remain specific, and very few, tables [12:23:17] quite. it will never be 'all tables', and will always be 'one at a time' addition process. [12:23:39] ok [12:24:28] technically it should be possible. we'd need to review capacity, and also i'd probably stonewall until https://bugzilla.wikimedia.org/show_bug.cgi?id=67450 is done [12:25:55] springle: makes sense, yeah. I'll talk to halfak and others and get back to you :) [12:26:09] cool :) [12:26:19] springle: batching inserts with at least some buffering would probably be a bit of work tho, esp. if we want to make sure we don't lose too much data [12:26:30] no doubt [12:26:38] but it won't scale much higher without that work [12:26:47] springle: indeed. [12:32:56] springle: also, unrelated, but is renaming a column in mysql always take forever? [12:33:19] (am not doing this in labsdb, just on my local mysql instance on quarry) [12:34:02] what version do you run? [12:34:47] i forget when that became an online operation, but definitely in mariadb renaming a column does not require a table rebuild [12:36:03] springle: mysql-5.5 [12:36:24] I could possibly just switch to MariaDB 10 [12:38:09] looks like mysql 5.6 would be needed for online ddl. so yeah, just switch :) [12:40:04] springle: hmm, we don't have 10 on apt.wikimedia.org [12:46:17] springle: and I can't seem to install from https://downloads.mariadb.org/mariadb/repositories/#mirror=somerset&distro=Ubuntu&distro_release=trusty&version=10.1 [12:46:18] hmm [12:52:14] YuviPanda: what breaks? [12:53:56] springle: The following packages have unmet dependencies: [12:53:56] mariadb-server-10.1 : Depends: mariadb-client-10.1 (>= 10.1.0+maria-1~trusty) but it is not going to be installed [12:53:56] E: Unable to correct problems, you have held broken packages. [12:54:58] (this is trusty) [12:55:53] https://mariadb.atlassian.net/browse/MDEV-5977 maybe [12:56:06] just guessing. i havn't used trusty yet [12:59:17] springle: sounds like it. ugh [12:59:31] springle: I'll have to live with 5.5 for now, I guess [12:59:45] no mysql 5.6 in trusty? [13:00:07] springle: not that I can see, no [13:00:31] springle: at least in main. I can probably install from oracle [13:00:47] springle: oh, there is [13:01:01] springle: but I'm already on mariadb 5.5, and I'm unsure what installing 5.6 will do [13:01:16] heh [13:03:10] springle: heh, upgrade works fine [13:03:28] * valhallasw`cloud pokes Dispenser [13:03:43] springle: boom, that was a quick alter table! [13:03:58] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND [13:03:58] 4274 dispense 20 0 662m 646m 644 R 47 16.3 350:40.91 metapixel [13:03:58] 22013 dispense 20 0 473m 457m 596 R 44 11.6 383:39.77 metapixel [13:03:58] 32639 dispense 20 0 186m 170m 480 R 39 4.3 38:51.87 metapixel [13:03:58] 10602 dispense 20 0 60308 42m 860 D 17 1.1 1:04.00 metapixel [13:04:02] Dispenser: killplz? [13:04:17] I thought Labs was powerful? [13:04:24] you're running it on tools-login [13:04:32] YuviPanda: nice. i'm always mildly surprised when stuff just works [13:04:39] springle: heh, me too [13:04:43] Where am I suppose to be running it? [13:04:48] the grid [13:05:09] jsub ALL the things [13:05:16] !jsub [13:05:22] valhallasw`cloud: Dispenser killed [13:05:27] * valhallasw`cloud always forgets what character the bot wants [13:05:36] But I can't monitor "The Grid" [13:05:51] in what sense? [13:05:51] I'm still setting it up [13:06:45] Dispenser: you should probably kill [13:06:46] 5967 dispense 20 0 4404 376 324 D 17 0.0 39:33.14 sh [13:06:47] as well [13:07:00] thanks [13:07:14] That one is running just fine [13:07:33] it was using 20% CPU, and isn't now [13:08:19] Dispenser: and if you want to test things with processes that take much memory / CPU, please use tools-dev and not -login. -dev has more RAM, etc [13:08:29] ok [13:09:29] Dispenser: but don't 'run' your tools there for real either :) as valhallasw`cloud said, use the grid [13:10:17] As if anyone's properly explained how it in the past 5 years [13:10:44] Dispenser: it was already required on the toolserver, and explained on the toolserver, and explained in the tools labs docs again [13:10:51] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Submitting.2C_managing_and_scheduling_jobs_on_the_grid [13:18:55] YuviPanda & springle. Read through scrollback and batch inserts bug. Will discuss how to get prioritized with analytics. Thanks for taking a look. [13:20:53] OK if I copy your conversation to the public analytics list? [13:20:59] YuviPanda, springle ^ [13:21:05] Not sure about the norms for that [13:21:05] * valhallasw`cloud waves at halfak [13:21:28] valhallasw`cloud, o/ [13:21:41] halfak: sure [13:21:47] halfak: it's publicly logged here, so should be ok [13:22:07] Great [13:24:00] I ran crontab on tool labs, and it is sending email like http://pastebin.com/QW4Sny45 . I use pywikibot and encording is set to UTF-8 (If you wonder, it is running on Korean Wikipedia) [13:24:36] Revi: that looks about right. [13:25:14] Revi: cron does not provide any encoding information in it's emails, so your email client is guessing (wrongly) that it's latin-1. Tell your mail client it's utf-8 and it should look more familiar :-) [13:25:19] Hmm I hoped it send email in correct word... not in broken words [13:25:25] Gmail? [13:25:50] * Revi googles [13:26:01] click the v next to the 'forward' button and click 'message text garbled?' [13:28:52] Revi: alternatively, send the mail manually by piping the output to 'mail', but that's also not completely trivial [14:13:53] hey andrewbogott, got a minute to help me with something? [14:13:59] (nothing scary, i promise) [14:19:29] ori: Sure, what's up? [14:20:52] andrewbogott: i need to add a service alias in beta -- 'udplog' should resolve to i-0000010b.eqiad.wmflabs / deployment-bastion / 10.68.16.58 [14:21:33] Does 'service alias' differ from a normal DNS entry? [14:21:38] i see the dnsmasq aliases in openstack.pp include a private and a public ip -- but this particular alias doesn't need a public ip. it doesn't hurt having one, but i'm blocked in that direction too, since it has all ips allotted [14:21:43] no, it's not -- it's the same [14:22:10] "since it has all ips allotted" -> since the project has used up its public ip allotment, i mean. [14:22:17] Giving it a public IP is easiest, I think. Otherwise it would involve tinkering with ldap or puppetizing something in /etc/hosts. [14:22:24] nod, i thought so too [14:22:26] I can just add another IP to the project if that gets you what you need. [14:22:52] yes please. then i'll submit a change to openstack.pp to add the alias [14:23:10] i saw recently that you need to restart dnsmasq on virt1001 to make it stick or something? is that something you could do? [14:24:05] hm, well, maybe I'm confused again :) [14:24:27] Is it going to be udplog.beta.wmflabs.org or something else that's fully.qualified.org? [14:24:43] I don't understand why this can't all be done via the normal 'manage address' web interface [14:25:36] andrewbogott: perhaps it could, actually [14:25:51] andrewbogott: i just can't add a hostname without a public ip, since the instance doesn't appear in the addresses interface otherwise [14:25:53] * ori tries [14:26:20] petan: around? [14:26:27] I raised the quota. So I think you can just allocate that new IP and point it… wherever. [14:26:44] petan: have you tried rcstream since it was fixed? [14:28:42] andrewbogott: hm, i can't add a .wmflabs hostname using that interface [14:28:50] andrewbogott: only udplog.wmflabs.org [14:28:59] what is the full address you want? [14:29:05] but i need a bare 'udplog' to resolve to the right instance in the project [14:29:40] Ah, see, that was my question about 'fully qualified' earlier... [14:30:01] So, in that case a public IP doesn't really help us I think. [14:30:02] i suck at DNS, sorry [14:30:14] Hm... [14:31:18] i think that $nova_dnsmasq_aliases in openstack.pp is where this needs to go, no? [14:33:20] hm, yeah. I guess I didn't know that was being used for that. [14:33:29] And I don't like it, although there's precedence. [14:33:49] Why not just /etc/hosts I wonder? [14:34:05] Anyway, copying the pattern that's already in that file should work OK. [14:35:27] * ori nods [14:35:40] andrewbogott: https://gerrit.wikimedia.org/r/#/c/153800/ [14:36:40] why does it have the same public IP but a different private IP from tools-webproxy? [14:37:41] andrewbogott: because i can't tell the difference between a '3' and a '9' apparently. sec [14:38:00] oh, ok :) [14:38:16] sorry :/ [14:38:24] amended [14:38:29] well, actually, I'm confused by this… bastion is at 208.80.155.191 right? [14:38:52] And yet that code refers to 208.80.155.131 which isn't even defined in the web interface. [14:38:55] So, what the heck is 208.80.155.131? [14:39:10] oh, wait, nm, that's for tools. [14:39:14] unrelated! [14:39:18] aye [14:40:09] ok, now I understand. [14:40:16] let's see what it does :) [14:41:51] andrewbogott: on https://gerrit.wikimedia.org/r/#/c/152791/ i saw: [14:41:58] "After merging this, dnsmasq needs to be restarted on virt$SOMETHING as part of the OpenStack network service. The last time this caused a bit of a boo-boo, so if in doubt, ping abogott for this. " [14:42:09] * ori was in doubt; pinged abogott for it. :) [14:42:24] yeah, I'll do that. not there yet... [14:42:43] ok, try now? [14:43:17] try what? resolving udplog? didn't work when i tried just now [14:43:23] ping: unknown host udplog [14:43:53] it needs to be udplog.eqiad.wmflabs somehow [14:44:26] is that different from the other aliases defined in that .pp? [14:44:38] i don't know; i didn't think so [14:45:15] tools-webproxy is indeed tools-webproxy.eqiad.wmflabs [14:45:21] so maybe it's a matter of waiting for the change to propagate? [14:56:49] ori: Well, everything is up to date and I've restarted nova-network. Too bad the people who did this before didn't actually explain what needs doing :( [14:57:23] also, for what it's worth, I hate that those things are defined labs-wide rather than locally to their project :( [14:58:43] andrewbogott: yes, amen [15:25:26] 3Wikimedia Labs / 3tools: Provide namespace IDs and names in the databases similar to toolserver.namespace - 10https://bugzilla.wikimedia.org/48625#c41 (10Marc A. Pelletier) 5REOP>3ASSI Renaming without prefix makes it easy to keep maintaining with the current tool. Just say the word, and I'll do the re... [15:26:29] * Coren desperately tries to swim upstream on the mail/bugzilla activity since he left for London. [15:28:23] ori, how would you feel about accomplishing this via /etc/hosts? That would get what you want, wouldn't it? [15:29:04] Otherwise I don't know how to make this work when there's no actual instance named updlog [15:29:04] andrewbogott: yes, sure. i don't mind that at all. i had no idea this would suck up so much time, sorry. [15:30:05] that would provide us with a wedge to rip out those other aliases too maybe… although at this point I'm not sure I understand what those dnsmasq aliases are doing at all. [15:30:36] * Coren hates dnsmasq with a passion usually reserved for serial killers and puppy kickers. [15:30:51] I still want to have real DNS in Labs. [15:50:11] ori: andrewbogott I dont think the 'udplog' entry in dnsmasq is going to work. [15:50:30] i'll just branch on $::realm [15:50:38] unless we use the production fully qualified name, that will resolve to the public IP [15:50:40] and we'll have something more elegant when we move to hiera [15:50:43] or not [15:50:47] hmm I am confused myself now [15:51:18] gotta run [16:55:26] 3Wikimedia Labs / 3tools: Include imagescaler::packages in tool labs - 10https://bugzilla.wikimedia.org/66354 (10Tim Landscheidt) 5PATC>3RESO/FIX [17:05:25] Coren: I'm fixing up the default sudo policies for new projects. Right now the default is dumb, it allows 'sudo as' from all members -> all members but doesn't allow regular old 'sudo'. [17:05:42] So, I'm wondering, should the default be that both things are allowed (sudo and sudo-as) or just sudo and leave sudo-as to be configured later if needed? [17:06:16] I'm inclined to the latter, but maybe I'm missing out on a case where 'sudo as' is always needed right away... [17:06:48] andrewbogott: Sudo to root alone just makes it possible but more annoying to sudo-as-user; so arguably if you have that you might as well always have the other. [17:06:56] (I.e.: sudo sudo -u someuser) [17:07:01] hm, true. [17:07:11] Ok, easy enough! [17:20:42] (03PS1) 10Ragesoss: Initial commit of current course stats tool [labs/tools/coursestats] - 10https://gerrit.wikimedia.org/r/153825 [17:22:01] (03CR) 10Ragesoss: [C: 032 V: 032] "initial commit of what's running at http://tools.wmflabs.org/coursestats/" [labs/tools/coursestats] - 10https://gerrit.wikimedia.org/r/153825 (owner: 10Ragesoss) [17:57:25] andrewbogott: wanna +2 https://gerrit.wikimedia.org/r/#/c/153600/ and https://gerrit.wikimedia.org/r/#/c/153784/? [19:13:08] any news on the issue with the webservice aborting randomly and silently? [19:14:40] ireas: typically out of memory [19:14:54] check lighttpd.err / qstat -s z [19:16:34] valhallasw`cloud: lighttpd.err does not exist, and qstat -s z probably only works if the webserver is not restarted yet? [19:16:49] at least, I don’t get any output after restarting the werbservice [19:17:06] qstat -s z should give you info on stopped jobs [19:17:14] not sure what happens after restarting the webservice... [19:17:19] you can still try qacct -j lighttpd [19:17:30] which should give info on all lighttpd jobs you've ever run [19:18:28] ireas: that'd be 'error.log' not lighttpd.err [19:18:36] Is it allowed to use https://pypi.python.org/pypi/goslate ? It is under a free license but accesses a nonfree API. [19:19:05] huh: That probably fails the open /data/ requirement. Lemme check. [19:19:07] error.log: [19:19:08] 2014-08-13 18:42:29: (server.c.1512) server stopped by UID = 0 PID = 27837 [19:19:11] 2014-08-13 19:02:54: (log.c.166) server started [19:19:38] hi Coren! already back at home? [19:19:51] ireas: Yeah, though not really "working" today. [19:19:59] by the way, I really enjoyed your keynote about the coolest labs projects :-) [19:20:16] huh: Actually, so long as you follow the TOS of the service that should be okay. [19:20:20] ireas: Thanks. [19:21:01] # Google forbits urllib2 User-Agent: Python-urllib/2.7 [19:21:01] request = Request(url, headers={'User-Agent':'Mozilla/4.0'}) [19:21:03] * valhallasw`cloud shudders [19:21:27] I don't know if it breaks the TOU or not [19:21:28] * YuviPanda|groggy pats Coren [19:21:32] Coren: nice meeting you in person :) [19:21:35] again [19:22:05] huh: it probably does [19:22:19] valhallasw`cloud: `qacct -j lighttpd` --> error: job name lighttpd not found [19:22:28] it doesn't use the official Google Translate API (which has a separate TOS and is not free/gratis), but rather the one used by browsers [19:22:45] how should I interpret ‘server stopped by UID = 0 PID = 27837’ (error.log)? [19:23:10] huh: yeah, I'm pretty sure that breaks Google's ToU [19:23:11] huh: right, which 99% sure you're not allowed to query automatically [19:23:20] ireas: Either you shut down the lighttpd or SGE did 'cause of OOM. What's maxvmem for that job? [19:23:38] ireas: As a kill by either a sysadmin or the grid itself; almost certainly out of vmem if the latter. [19:25:32] scfc_de, Corent: it is a regular webservice executing Python files (some directly, some using FastCGI) -- can I manually adjust the memory or detect the memory leak? or could it be a bug in the webservice? [19:27:19] https://support.google.com/webmasters/answer/66357?hl=en [19:27:38] if it breaks Google's ToU not sure why it is allowed on python.org [19:27:41] ... [19:28:07] ireas: The webservice's memory limit can be increased, but "4 GByte ought to be enough for anyone" :-). I don't know how to usefully debug that, though. [19:28:47] scfc_de: yes, I agree regarding the 4G ;) [19:29:04] huh: because the ToU are between you and google, not between python and google? [19:29:17] also I don't think anyone checks packages on pypi [19:29:34] so you probably can't even assume you're allowed to use them ;-) [19:30:04] ireas: Tracking down leaks tend to be really hard, especially if they are data-driven. One of the most common causes however is reading large result sets in memory from python. [19:30:23] ireas: Do you have database queries in your tools that can, under some circumstances, return lots of rows? [19:32:52] Coren: the two queried tables could return 50k rows. I’ll have to check the source code for the queries, but they should never query more than ~ 100 rows. [19:33:17] I’ll have a look at that and then ask again. ;-) [19:35:15] Coren: we need better toooooooooooooling :-p [19:36:04] valhallasw`cloud: I usually ssh into the exec hosts and strace them :P [19:36:08] I don't know if non roots can do that [19:36:30] I would be surprised if non-roots can strace :-p [19:37:38] maybe a ulimit call? Then at least it crashes with a memory error, although probably not at the most sensible point [19:38:27] You can strace your own processes, but that's of limited usefulness unless you already have a good idea what to look for. [19:42:29] ireas: as a tip: include a counter in your pathon fcgi, one it reaches lets say 500 calls, let it exit(). [19:42:50] lighty will respawn a new, fresh one. all leaks gone [19:45:27] doesn't lighttpd do that by default? [19:45:52] valhallasw`cloud: only for php-fcgi [19:46:30] ugh. [19:47:06] Ihave a python.fcgi that accumulates from 800M to 1g to 2g, don't know why? I detroy it once in a while and everything starts over - fresh [19:48:54] hedonil: thanks; I’ll keep that in mind [19:50:35] ireas: yeah, it's a mystery, tried a lot of things, but gave up eventually, exit/kill & auto-respawn it is [19:52:12] It's entirely possible that python /itself/ has a leak, or whichever libraries you are using through no fault of your own. Many, many libraries are written with no solid testing on scale or liftetime, and garabage-collected larnagues are very vulnerable to large classes of memory leaks with no easy fixes. [19:53:06] (cyclic reference graphs; badly-scoped dynamic objects; "behind the scenes" allocation from other libraries; ownership of objects, etc) [19:54:26] I'm just running some tests with opcode cacher XCache and lighty with php-fpm, will report when done [19:55:57] Coren: yeah, plain cgi isn't that bad ;) - as said, a counter & exit is a good thing that clears up everything in python while your webservice keeps up [20:00:07] hedonil: This is why I like languages that allow me to manage memory (or any resource, really) myself. If I gots a leak at least it's /my/ fault and I can fix it. :-) [20:00:49] Coren: unless you use a third party library and you're back where you started [20:00:58] Coren: true, but not in everyone's foo ;-) [20:01:37] valhallasw`cloud: Not really; it's easy to constrain what resources random libraries eat iff you have lower-level access to the allocation system. [20:02:20] Coren: err? You want to patch malloc()? [20:03:04] valhallasw`cloud: I often replace it with a per-request arena allocator when I know I have to be stringent about resources (embedded stuff, HR coding) [20:03:20] Hm. Interesting. [20:04:43] valhallasw`cloud: Also, given a programming language with explicit memory allocation, the libraries that do not give you pointers back to everything it allocated are clearly ill-behaved and need to be fixed (and unfailingly are) [20:05:58] It's "more trouble for the programmer" up until you run into trouble. :-) [20:18:41] 3Wikimedia Labs / 3deployment-prep (beta): Beta cluster job queue not running - 10https://bugzilla.wikimedia.org/69272#c14 (10Aaron Schulz) Can this be closed now? [20:37:49] Coren: gonna check this out right now [20:37:51] http://new.livestream.com/accounts/9511962/events/3254466/videos/58723969 [20:40:42] 3Wikimedia Labs / 3deployment-prep (beta): HTML Tags are stripped from extension output - 10https://bugzilla.wikimedia.org/66516 (10physikerwelt) [20:53:03] hedonil: I've been told I didn't suck. [20:54:55] Coren: are there slides online somewhere? [20:54:56] 3Wikimedia Labs / 3deployment-prep (beta): HTML Tags are stripped from extension output - 10https://bugzilla.wikimedia.org/66516#c2 (10physikerwelt) 5UNCO>3NEW I could reproduce the bug locally by setting $wgUseTidy=true; It seems to be tidy bug reported as http://sourceforge.net/p/tidy/patches/84/ b... [20:55:12] valhallasw`cloud: They're on Commons. Lemme dig the url for you. [20:55:49] Found it! https://commons.wikimedia.org/wiki/File:Showcase_ALL_the_cool_things_(WM2014_Presentation).pdf [20:55:51] thanks [21:02:33] Coren: I can confirm, you didn't ;-) [21:03:32] really cool stuff, presented confidentially [21:04:17] \o/ to all [21:05:47] ... "confidentially"? I expect you used the wrong word by accident there because I'm pretty sure a presentation on the featured track at the biggest Wikimania yet that was also broadcast live is not very confidential. :-) [21:06:21] hehe [21:06:41] /ly/d [21:06:50] 'with confidence' [21:07:00] Oh! Ah! Thanks. [21:07:16] confident or sovereign [21:07:57] !log deployment-prep updated OCG to version faee29a260a96cbc0cdc0d402658b754a7425af9 [21:08:04] confidently? I guess :-p [21:08:16] valhallasw`cloud: :P [21:08:46] it's the keyboard... [21:09:46] A good promotion for Tool Labs anyway [21:10:25] .. and its unleashed creativity [21:11:39] It's my first "soft" presentation where I have, bascially, zero technical content. I was actually scared out of my mind. :-) [21:15:34] Coren: is https://wikitech.wikimedia.org/ DNS broken? [21:15:52] dafu? Looking into it. [21:16:06] Reedy says it's not DNS but known (on -tech) [21:16:34] looks like https, not dns? [21:16:50] (http:// works and forwards to https:// which doesn't work) [21:17:04] http://wikitech.wikimedia.org/ doesn't work for me [21:17:14] Coren: what's listening to wikipedia tools name? [21:17:43] now it works [21:19:23] huh: Yeah, it was a puppet issue which others have just fixed. [21:19:32] hedonil: It's not running on Labs at all. [21:19:51] hedonil: http://listen.hatnote.com/ [21:20:09] Coren: ah thx [21:22:14] fancy-pancy =-O [21:33:38] weird! bookmarked. [22:45:31] 3Wikimedia Labs / 3deployment-prep (beta): The Salt minion client id should be the FQDN, not ec2_instance_id - 10https://bugzilla.wikimedia.org/69502 (10Ori Livneh) 3NEW p:3Unprio s:3normal a:3None There's no reason for the minion id to be set to the old instance ID rather than the FQDN. [23:01:43] 3Wikimedia Labs / 3Infrastructure: wmflabs changes to https:// after account creation (Safari) - 10https://bugzilla.wikimedia.org/69504 (10Kristen Lans) 3NEW p:3Unprio s:3minor a:3None To reproduce: -Open Safari (I used v7.0.3) and go to http://en.wikipedia.beta.wmflabs.org/w/index.php?title=Special...