[00:15:58] ori: Haven't started it yet. Starting tomorow. [00:16:10] ah nod [02:57:11] ori: Did you know there used to be a perlbal? (before pybal). Just noticed it mentioned randomly in a SAL entry by TimStarling from 2005. Around the time we backed up things to a Seoul, Korea data centre. [02:57:27] https://wikitech.wikimedia.org/wiki/Server_admin_log/Archive_5#October_28 [02:57:50] perlbal was an HTTP proxy written in perl [02:57:58] not really a pybal predecessor [02:58:14] Aha [02:58:20] TimStarling: What was it used for? [02:58:23] the initial LVS implementation used a PHP monitor script called lvsmon, written by me, that was the predecessor to pybal [02:59:54] well, it was used for things that LVS is used for now [03:00:12] Right, load balancing by proxy rather than at the TCP level [03:00:21] those logs are refreshing my memory [03:01:06] I don't think it was used for frontend traffic, I think we used DNS round robin for that [03:01:41] https://upload.wikimedia.org/wikipedia/commons/8/87/Wikimedia_Servers.svg#2005 [03:02:27] before that load was balanced by quickly plugging and unplugging an ethernet plug between several servers [03:04:52] wasn't DNS round robin how mark got involved? [03:05:14] I think mark got involved when we started doing geodns [03:05:20] "Tim: Deployed pound/icpagent on dalembert. It is currently running alongside perlbal instances on bacon and holbach." [03:05:43] I think we tried perlbal and pound for a while as a replacement for gwicke's original system which was based on icpagent [03:06:39] icpagent was a hacked up squid which did nothing but respond to ICP requests [03:06:57] ICP being inter-cache protocol [03:07:58] squid's implemented ICP such that when a frontend request came in, it would send out ICP requests to all its backends [03:08:14] and then the first to respond would be used as the backend for that request [03:08:32] so gwicke hacked up squid to make icpagent, a dedicated ICP responder which ran on the apaches [03:09:12] the idea was that an apache under load would have a slightly slower ICP response time and so would be deprioritized [03:09:15] so that the apaches could function as squid backends? [03:09:35] yes, I mean, you can just turn off ICP but this was a way to use ICP as a load detector [03:10:11] I didn't like it for some reason, I think it was quite difficult to control, and gwicke was gone by then anyway [03:10:42] we trialled perlbal and pound as replacements, with icpagent running on the perlbal/pound host [03:10:49] with the ICP configuration still active, we had an easy fallback [03:11:01] we could just tune the ICP delay to send all traffic to perlbal or pound [03:11:28] but it was slow, and both were pretty rubbish, so it wasn't long before I introduced LVS [03:11:55] I think perlbal was used for a while after that as a search backend [03:12:08] before we eventually put search on LVS as well [03:12:18] latency as configuration [03:12:45] yeah [03:13:09] so now you understand "noticed that perlbal was still taking a fair bit of load on bacon. Killed icpagent on bacon, increased icpagent delay time from 1 to 5ms on holbach." [03:13:29] Yeah, I was wondering about "bacon" [03:14:07] bacon was running perlbal but it was meant to be deprioritised, with an icpagent delay [03:14:11] Not surprisingly, that line is the excerpt of the first google for hit for https://www.google.co.uk/search?q=bacon+perlbal [03:14:29] but I noticed it was still getting traffic, so I just killed icpagent on it, which stops squid from sending it requests [03:14:40] Krinkle: that should be a dish [03:14:52] right, the ping would reach a deadend [03:14:53] and I increased icpagent delay time on holbach which was probably another perlbal backend [03:14:55] "21:49 Tim: LVS now in service in front of yaseo apaches. yf1010-1017 are now in service as apaches" [03:15:35] this seems like a really horrendous scheme [03:16:02] what was yf? was yaseo basically using the same numbers (1000-1999) as are now used by eqiad? [03:16:04] looks like at the time, bacon was being used for backups [03:16:45] so it was removed from perlbal service to make the backups go faster [03:16:52] there was a lot of sharing :) [03:17:04] there are some good ideas re load balancing in http://www.cs.duke.edu/courses/compsci590.4/fall13/838-CloudPapers/dean_longtail.pdf [03:17:14] TimStarling: Hm.. so yaseo wasn't backup, they were primary for certain databases/ certain wikis. [03:17:31] they were primary for zhwiki and kowiki [03:17:41] like, database and all. [03:17:44] interesting [03:17:53] the hostnames were assigned by yahoo, I don't remember what "f" stood for but I don't think we set that [03:18:04] yeah, I set it all up [03:18:36] there were lots of pops around that time [03:18:39] Asia now goes through ulsfo, right? Which for some bizarre reason is "closer" than esams [03:18:42] I remember another one in paris [03:18:51] with really old machines that were donated [03:18:56] In terms of latancy [03:18:57] pretty sure ulsfo is closer [03:19:09] Yeah, but what route does it take? Through russia? [03:19:10] I didn't like it for some reason, I think it was quite difficult to control [03:19:16] (gwicke or the tool? :P) [03:19:18] gwicke, lopar [03:19:25] but seriously, not drawing a hard and fast distinction between accidental network conditions and deliberate server configuration seems a little crazy [03:19:41] "is this server depooled or just far away?" [03:19:43] gwicke: https://wikitech.wikimedia.org/wiki/Clusters#Past :) [03:19:52] asia to ulsfo? straight across the pacific [03:20:09] ocean cables are cheaper than land cables [03:20:12] Oh, right. [03:20:17] Just like US-EU [03:20:26] I didn't think there were cables on the other side as well [03:20:27] Krenair, Krinkle: indeed [03:20:27] makes sense [03:20:30] where does it enter? [03:20:37] undersea cables just sit on the seafloor, land cables need to be buried [03:20:50] http://www.submarinecablemap.com/ [03:20:59] the ships that lay them are awesome [03:21:10] so eventually it was decided to move stuff back into pmtpa? or did pmtpa not exist at that point? [03:21:28] they have a giant spool on deck [03:21:49] it looks like the San Diego 'cluster' didn't qualify for that wiki page [03:22:16] the yaseo servers were always owned by yahoo [03:22:25] (ok, looks like pmtpa existed then) [03:22:35] There is only one mention of San Diego on the whole wikitech [03:22:36] https://wikitech.wikimedia.org/wiki/Server_history [03:22:41] Krenair: it was SD -> pmtpa [03:22:50] early 2004 [03:23:15] yahoo effectively donated those servers to us, but they didn't expand the cluster [03:23:15] I only set up wikitech after that move [03:23:34] interest from them died off, mark had a hard time getting them to talk to us [03:23:58] we basically outgrew their initial donation and they weren't interested in making another one [03:24:39] so yes, we reintegrated with pmtpa [03:25:05] lopar was a bit of a joke, just three celerons [03:25:25] IIRC SD was two boxes [03:25:46] they donated bandwidth but they had separate costs for transit and peering [03:25:49] probably more bomis than wp back then [03:26:03] and they were very alarmed at the transit costs we were causing for them [03:26:05] Krinkle: http://www.slate.com/blogs/future_tense/2014/08/15/shark_attacks_threaten_google_s_undersea_internet_cables_video.html [03:26:09] even with only three celerons, hehe [03:29:46] There is only one mention of San Diego on the whole wikitech [03:29:55] presumably because San Diego predates wikitech [03:30:46] by the time I got involved, no Bomis staff were left in San Diego [03:31:04] hehe, that's before Gabriel said it predated Wikitech. Makes sense [03:31:18] and when one of the RAID 0 database servers went down due to a broken disk or something, Jason Richey had to drive from near LA to go fix it [03:32:15] http://marc.info/?l=wikipedia-l&m=105291315821915&w=2 [03:33:16] when was yaseo donated and when did it stop being used? [03:33:52] http://wikitech-l.wikimedia.narkive.com/rLQ5UTJf/opteron-s-here-to-battle-the-evil-forces-of-the-decepticons [03:35:03] Krenair: well, you can see from the server admin logs [03:35:31] first mention of yaseo is on September 10, which shows jeronim installing operating systems [03:36:34] I saw that but it was reinstalling [03:37:56] yeah but there was nothing before that [03:38:12] he probably means that they had operating systems when they were given to us [03:41:16] if anyone has IRC logs of the relevant channels from before 2008 I would love to have a copy [03:42:50] I might have some [03:43:23] will have a look next time I get my old backup disk out [03:43:38] https://wikitech.wikimedia.org/wiki/Server_admin_log/Archive_3#16_February [03:43:55] Did the default wgDisableAsksql just open up raw SQL access to the world or something? [03:44:04] https://wikitech.wikimedia.org/wiki/Obsolete:Yaseo_migration_plan [03:44:16] Yep, Special:AskSql [03:44:24] I cleaned up some left-over of it in 2010 [03:44:32] yeah [03:45:02] you know, one of the things I most liked about getting shell access in 2003 was that I was able to log in and kill my runaway queries when I accidentally took down wikipedia with Speical:Asksql [03:45:28] lol [03:46:44] of course, write access was not allowed, and access to passwords and what not was not allowed [03:46:56] it was a special MySQL user with restricted permissions [03:47:22] still very easy to take the site down by accident though, especially when you were just learning SQL [03:47:24] Looks like wikipedia.tv (mentioned in one of the mailing lists from [[Server history]]) is still owned by @yahoo someone in Netherlands [03:49:31] also only admins were allowed to use it [03:51:02] was this when it was admin-by-shared-password? [03:51:07] no [03:51:19] you should see what I had to go through to get admin access, you think RFA is bad [03:51:32] http://article.gmane.org/gmane.science.linguistics.wikipedia.english/2121/match=tim+starling [03:52:13] hehe, "I hardly ever get into POV disputes or edit wars of any kind" [03:52:21] http://article.gmane.org/gmane.science.linguistics.wikipedia.english/2133/match=tim+starling [03:52:23] Krenair: Which source for 2005 and 2009 of yaseo? [03:53:02] https://wikitech.wikimedia.org/wiki/Server_admin_log/Archive_5#September_10 [03:53:28] https://wikitech.wikimedia.org/wiki/Server_admin_log/Archive_14#July_23 [03:57:28] thx [04:03:17] I found phase3_cvs and phase3_tim folders, but no irc logs yet [04:12:08] https://office.wikimedia.org/w/index.php?title=Bash&type=revision&diff=156482&oldid=156015 :) [04:13:09] not https://www.mediawiki.org/wiki/Quips ? [04:13:25] or https://tools.wmflabs.org/bash/about [04:13:30] .. [04:14:59] !bash you know, one of the things I most liked about getting shell access in 2003 was that I was able to log in and kill my runaway queries when I accidentally took down wikipedia with Speical:Asksql [04:15:04] (I think that's how it works?) [04:15:37] stashbot_ is here [04:16:39] would have been nice if you corrected my spelling ;) [04:17:12] s/Speical/Special [04:18:57] I found some logs from 2004 [04:20:33] yay, thanks gwicke! [04:22:52] I definitely have #mediawiki [04:22:59] any other channels you are interested in? [04:23:56] upload all of them [04:24:05] they're valuable documents [04:24:09] * ori isn't kidding [04:24:23] https://archive.org/ ! [04:24:57] TimStarling: very well. fixed ;) [04:25:19] #wikipedia [04:25:56] well, some of those channels had a no public logging policy [04:26:01] and archive.org doesn't do private archives [04:27:58] there was no implicit or explicit guarantee of privacy either, and the historical value trumps these concerns [04:30:16] we even made a fuss when someone posted public logs, we threatened them with banning [04:31:42] there's a lot of stuff that I would like to be preserved for publication in 50 years or so, but 10 seems a bit short [04:36:12] TimStarling: I shared a tar via google drive [04:36:25] I included some private conversations [04:37:54] got it, thanks [04:39:06] I don't think there is anything sensitive in there, but didn't check either [04:44:41] the #wikipedia topics at least don't say anything about logging [04:45:34] were you on #wikipedia-cabal? [04:47:15] don't think so [04:47:32] was that an admin channel? [04:49:53] good night folks! [04:51:33] I'm heading out as well [04:51:43] bye! [04:54:27] cya [05:25:11] [20:41:17] if anyone has IRC logs of the relevant channels from before 2008 I would love to have a copy <-- which channels? [05:25:54] all wikimedia-related channels [05:28:09] my IRC logs only go back to April 2008 [05:29:59] I probably have some 2007 logs on a backup drive, that computer died 5 years ago [05:33:46] https://tools.wmflabs.org/bash/quip/AU_ZgzcB1oXzWjit5ktB fixed the typo :P [05:34:33] hard drives don't last forever, you should probably migrate them if you want to keep them [05:34:52] but I suspect there are a few people with logs of everything, going back a long way [05:35:19] Erik Moeller probably has some [05:39:05] my logs would be pretty incomplete, I didn't have a bouncer at that time, and I would usually sign on from random school computers [05:43:38] yeah, never mind then [18:41:30] ori: can you give https://gerrit.wikimedia.org/r/#/c/199598 another go? [20:28:12] anomie: do have any time to look at https://gerrit.wikimedia.org/r/#/c/205528/ ? Kunal is super busy atm [20:38:47] tgr: yes [20:38:53] tgr: do I have your permission to make cosmetic fixes? [20:57:26] ori: sure [20:58:38] AaronSchulz: is "CAS" in the context of your patch the CAS protocol, or are we reusing an acronym because NQE TLAs [20:58:48] compare and swap [21:03:00] tgr: does pg_dump's --table argument indeed accept wildcards? [21:03:03] ('sentry_*') [21:03:14] yes [21:03:17] cool, just checking [21:06:41] tgr: when initializing the sentry database, you call sentry_cli with '--noinput'; when creating the superuser, you call it with '--no-input'. which is it? [21:15:04] ori: that's correct, sentry is inconsistent [21:57:56] anomie: the last PS in the password handling and auth hook patches was just a rebase, right? [22:12:10] also for the main authmanager patch