[16:22:09] using php5 [16:22:09] Process token [0]: T_INLINE_HTML => #!/usr/bin/php\n [16:22:21] hhvm [16:22:25] Process token [0]: T_HASHBANG => #!/usr/bin/php\n [16:25:57] 10Deployment-Systems, 6operations, 7HHVM, 15User-Bd808-Test: Scap should restart HHVM - https://phabricator.wikimedia.org/T103008#1382863 (10Joe) [16:28:22] yay [16:28:27] fixed in HHVM 3.7: https://github.com/facebook/hhvm/commit/3ab5dce03b1420562e1c0d6ba8f7b3433289e695 [16:36:44] (03PS2) 10Legoktm: Update HISTORY.md and README.md for 0.3.0 release [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/219390 [16:36:46] (03PS1) 10Legoktm: CharacterBeforePHPOpeningTagSniff: Support T_HASHBANG for HHVM >=3.5,<3.7 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/219394 (https://phabricator.wikimedia.org/T103119) [16:37:25] (03PS3) 10Legoktm: Update HISTORY.md and README.md for 0.3.0 release [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/219390 [16:48:47] !log primed keyholder on deployment-bastion [16:48:50] Logged the message, Master [16:49:26] it's weird, I can't seem to login to deployment-cache-text02 via my ssh key or via keyholder. [16:50:49] 10Continuous-Integration-Infrastructure, 10Gerrit-Migration, 3releng-201516-q1: Prototype CI integration with Differential - https://phabricator.wikimedia.org/T103127#1382950 (10greg) 3NEW [16:58:29] Yippee, build fixed! [16:58:29] Project beta-scap-eqiad build #57900: FIXED in 4 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/57900/ [17:17:28] (03PS1) 10Legoktm: Create generic 'doxygen-publish' job for PHP packages [integration/config] - 10https://gerrit.wikimedia.org/r/219401 (https://phabricator.wikimedia.org/T102974) [17:32:49] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Zuul: Zuul-cloner should use git-clean to reset workspace - https://phabricator.wikimedia.org/T76304#1383067 (10ori) a:5ori>3None [17:39:10] Dear CI haxors, I'm trying to debug a zuul/submodule situation, and want to get the treeish of different repos for my build. Can't we disable the "rm -rf $WORKSPACE" step somewhere, in https://integration.wikimedia.org/ci/view/WMF/job/wikimedia-fundraising-civicrm/configure perhaps? [17:39:29] Krinkle: ^ if you have a minute [17:42:21] ejegg: We could do something truly horrible like add a script which git log -p -1 > /tmp... [17:43:06] yeah.... i thought of that :P I already barfed all over the SmashPig commit log [17:43:07] or I think we could temporarily merge "drush -v -v" for the civibuildkit step which enables modules [17:43:26] oh right, lemme try -v -v locally to see if it's even useful [17:46:00] ejegg: even more evil: "mv $WORKSPACE /tmp/crm-debugging" [17:46:58] well, just out of curiousity, how do i log in to integration-slave-trusty-1014 ? [17:47:24] I... add you to the labs project [17:47:34] Though I think you already are, checking now... [17:48:04] ah, thanks! I have done shamefully little on labs for being here a year [17:48:24] The ultimate haze :) [17:49:49] ejegg: OK, you're added to the project as an admin... ahem. (<-- Krinkle) [17:50:26] oh hey, i have a *.wmflabs in my ssh config. Guess I at least intended to go there at some point. [17:50:53] are the integration slaves .eqiad.wmflabs? [17:51:21] ejegg: yup: integration-slave-trusty-1014.eqiad.wmflabs [17:51:36] thanks! [17:53:21] awight: why? [17:53:23] ejegg: I'm gonna uncheck "Discard old builds" just to see [17:53:41] Well that's a new one: got through the bastion, but then Connection closed by UNKNOWN [17:53:44] Discard old builds is about the logs saved in Jenkins [17:54:00] Krinkle: we're trying to debug a CI job, we're not convinced that zuul+submodules is doing the right thing. [17:54:24] Feel free to veto or downgrade our perms, btw, I just need login, not admin [17:55:25] awight: the extra -v doesn't add aything to the drush enable output (one -v just lists individual modules as they're done). And -d just adds timing [17:56:17] ejegg: Oh, that's really strange. for "drush updatedb", two -v flags totally emits a line for each migration [17:56:34] The job definition of wikimedia-fundraising-civicrm is missing any declaration of 'defaults' or 'scm' [17:56:38] drush? that sounds like Drupal? [17:56:47] where do we have that [17:56:48] mutante: yep :) [17:56:50] As such, it is entirely undefined how it will fetch and what is preserved. [17:56:58] * awight duct tapes mutante [17:57:00] ejegg: where do we run that? [17:57:02] :) [17:57:17] fundraising's using CiviCRM, which runs on top of it [17:57:37] Krinkle: wat. I was seeing "falling back to master", but that's a lie you're saying? [17:57:37] and we have all sorts of custom modules [17:57:40] gotcha, thx [17:58:08] fwiw /mnt/jenkins-workspace/workspace/wikimedia-fundraising-civicrm/src/wikimedia/fundraising is still on integration-trusty-slave-1014 [17:58:09] awight: No, I'm saying the job definition in confg/jjb is missing properties that every other job I know of does have declared. [17:58:13] So its behaviour is sketchy [17:58:23] pick a default you think makes sense and declare it. [17:58:29] We'll duct tape wordpress and joomla in as soon as we can figure out a place for them [17:58:30] e.g. shallow git clone, with or without submodules etc. [17:58:50] look for uses of "defaults" in other jjb jobs for examples [17:58:59] and defaults.yaml for the available options [17:59:29] mutante: Sorry if we triggered any PTSD from old jobs or anything ;) "drupal" would be a good halloween costume for an ops party, I'm sure [17:59:36] the only three relevant options in this case are use-remoteonly-zuul, use-remote-zuul-no-submodules and use-remote-zuul-shallow-clone [18:00:07] awight: you're explicitly listing all the repos to clone, so we don't actually need submodules, right? [18:00:12] right [18:00:20] awight: haha, i'm just glad we don't have 2 CiviCRMs anymore :) it's ok [18:00:38] the first two will preserve .git between builds to speed up the build (but may leave behind stuff), use-remote-zuul-shallow-clone is clean every build and is sped up by doing shallow clones (e.g. not infinite history but just the current HEAD) [18:00:40] mutante: I keep having this nightmare that we do [18:00:56] awight: it was true for a looon time, but not anymore :) [18:01:12] shallow-clone sounds like our best bet [18:01:13] since you need submodules, declare "defaults: git-remoteonly-zuul" [18:01:30] that is "defaults: use-remoteonly-zuul" [18:01:35] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 6Release-Engineering: Automatic account creation for MW-Selenium test user in integration environment - https://phabricator.wikimedia.org/T103135#1383255 (10dduvall) 3NEW a:3dduvall [18:01:44] Krinkle: I think we want to shallow clone, and explicitly check out the sub-repos we need [18:02:04] for example, here we're updating the crm/drupal submodule [18:02:12] Right [18:02:21] So you dont' want to use jenkins git at all [18:02:23] but want to use the crm master HEAD, which doesn't have its drupal pointer updated yet [18:02:29] but bypass and use zuul-cloner exclusively [18:02:34] which doesn't support submodules, or clean builds. [18:02:50] but will allow the separate repos you need [18:03:06] it also doesn't support shallow clones atm [18:03:46] ok, I see we have zuul-cloner: projects: > and the list of 5 [18:03:55] Yeah, that's god [18:03:56] good [18:04:13] should that be getting us the lastet master crm on every recheck of a drupal build? [18:04:33] it claims to in the log [18:04:35] It first tries a commit by the hash of the change that triggered the build [18:04:46] (which will match in teh repo that triggered it) [18:05:12] then it tries the target branch of the commit being triggered (e.g. a commit to REL1_24 wil check out REL1_24 of the other repos) [18:05:15] The thing I was worried about, was that we might be using stale .git directories for repos that were not the one under test. [18:05:18] if that doesn' exist, it uses master [18:05:21] so, if the drupal submodule is still at the same commit, it re-uses the whole batch of them? [18:05:30] even if crm has been updated since? [18:05:35] this 3-part logic is done for each repo separaetly. [18:06:18] It doesn't use submodules, they don't exist hee [18:06:20] here* [18:06:29] it only uses the repos you specify [18:06:47] each build it ensures each repo is either at the change commit, target branch head, or master branch head. [18:06:49] ahh, so that's why it says Project wikimedia/fundraising/civicrm-buildkit in Zuul does not have ref refs/zuul/master/Zd78d6ffdec9147219392154fa7cb945c [18:06:52] it won't go stale [18:06:58] b/c that ref is only in drupal [18:07:05] yeah, that's fine [18:07:08] we just enable verbose logging [18:07:25] then it falls back to HEAD and does get fresh crm etc [18:08:04] ok. in that case, i'd be curious to turn on full mysql logging for a build. [18:08:47] mysql? [18:09:01] but need to work out something with my login - getting 'Connection closed by UNKNOWN' when trying to get into integration slaves after getting past the bastion [18:09:16] awight: oh, right, are we using sqlite for tests? [18:09:18] thcipriani: so... we can bring up /data/project again now [18:09:21] thcipriani: if we want to :) [18:09:24] try logging into the bastion, then ssh -v to the node [18:09:24] thcipriani: I guess we do want ot [18:09:25] *to [18:09:33] thanks awight [18:10:20] YuviPanda: heh, yes, we do, there's even a phab ticket somewhere [18:10:30] thcipriani: I'll turn it back on now [18:10:35] thanks [18:11:59] thcipriani: actually can oyu do that? I'm in the middle of four hundred other things [18:12:56] YuviPanda: sure, what all needs to be done? [18:13:03] awight: ejegg: OK. I don't know what error you're debuggng, but letme know if you need anything. One thing you may wanna try is changing the job config and adding "rm -rf src/" inside the first execute step before it does zuul-cloner. [18:13:08] Just to make sure old stuff is gone [18:13:16] ejegg: Sorry, we're using mysql. Krinkle actually just switched backends from sqlite to mysql for even the MediaWiki tests. But I was just curious about the plan to log queries [18:13:22] it'll slow down the build since it'll re-clone each build, but just for debugging [18:13:45] thcipriani: so under nfs_mounts: [18:13:48] turn project to true [18:13:50] and run puppet [18:13:52] awight: just cos I can't get drush to output all the schema updates [18:14:30] Krinkle: one more question. Is there a way to disable the "rm -rf $WORKSPACE" step? [18:15:21] awight: Where do you see t hat? [18:15:24] That doesn't exist afaik [18:15:53] ahh hrm. I thought I had that set up for a while when I was writing the job definition. [18:15:56] k [18:16:38] but yes, if it is in there, just remove it via jenkins job/../configure page [18:17:09] and then retrigger the old build from the sidebar (no need to go through gerrit/zuul) [18:22:06] 6Release-Engineering, 6Labs, 6operations, 10wikitech.wikimedia.org: silver / scap - Could not get latest version: 403 Forbidden - https://phabricator.wikimedia.org/T103138#1383336 (10Dzahn) [18:36:35] !log added role::deployment::repo_config to deployment-prep hiera, to be removed after patched in ops/puppet [18:36:37] Logged the message, Master [18:38:33] (03CR) 10Polybuildr: [C: 031] CharacterBeforePHPOpeningTagSniff: Support T_HASHBANG for HHVM >=3.5,<3.7 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/219394 (https://phabricator.wikimedia.org/T103119) (owner: 10Legoktm) [18:39:45] !log running `salt -b 2 '*' cmd.run 'puppet agent -t'` from deployment salt to remount /data/projects [18:39:48] Logged the message, Master [18:40:22] thcipriani: yay :) [18:40:37] thcipriani: remember this is a 9day old backup [18:42:12] (03CR) 10Polybuildr: [C: 031] Update HISTORY.md and README.md for 0.3.0 release [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/219390 (owner: 10Legoktm) [18:46:57] Yippee, build fixed! [18:46:58] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #2118: FIXED in 56 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/2118/ [19:17:43] 6Release-Engineering, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, 6Multimedia, and 6 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1383626 (10Tgr) > Just tell people to trust their OS' CA store, anything else is just insecure. Telling people t... [19:20:38] 5Continuous-Integration-Isolation, 7Jenkins: Nodepool can't create slaves on Jenkins - https://phabricator.wikimedia.org/T103120#1383629 (10hashar) Found the same trace on a couple sources: https://github.com/arangamani/jenkins_api_client/issues/137 "Errors in creating a jenkins node" which ends up pointing t... [19:42:08] 5Continuous-Integration-Isolation, 7Jenkins, 7Upstream: Nodepool can't create slaves on Jenkins - https://phabricator.wikimedia.org/T103120#1383682 (10hashar) [19:43:23] 5Continuous-Integration-Isolation, 7Jenkins, 7Upstream: Nodepool can't create slaves on Jenkins - https://phabricator.wikimedia.org/T103120#1383686 (10hashar) p:5Triage>3Lowest I have reported the issue to upstream as https://issues.jenkins-ci.org/browse/JENKINS-28993 Anyway, the nodepool ssh key has t... [19:43:39] hashar: That is the longest stacktrace I have ever seen :o [19:43:47] polybuildr: welcome to java [19:43:50] and that one is stripped [19:44:03] ... 82 more [19:44:05] I saw. :o [19:44:19] in lot of java code, there is 3-10 lines of glue for each actual line of code executed [19:44:27] or everything is a method :-} [19:44:43] that produces somehow useful stack traces that let you find out the chain of execution. That is handy [19:44:54] I see. [19:45:02] anyway out [19:45:09] I've never written Java, but I have written C#. I don't remember the kind of stacktraces I used to see then. [19:45:16] o/ [19:46:22] * hashar waves [19:47:31] hey y'all [19:47:38] got a problem with beta cluster [19:48:30] this: baseUrl = mw.config.get( 'wgEventLoggingBaseUri' ); makes baseUrl = meta.wikimedia.org right now [19:48:34] in beta [19:48:45] which means all beta client side Event Logging events are ending up in prod... [19:49:05] could someone help me figure out how that was set incorrectly and why? [19:50:04] disregard ^ brain hiccup, never mind [19:50:12] don't know enough to help, but out of curiosity, example wiki please? [19:50:20] I thought test2 was on the cluster [19:50:22] and it doesn't happen there [19:50:31] okay :P [20:00:25] polybuildr: let me fiddle with gerrit and see if I can get you +2 [20:02:26] legoktm: okay :D [20:03:51] In the mean, I shall prepare for this immense responsibility by reading https://www.mediawiki.org/wiki/Gerrit/%2B2 [20:03:51] s/mean/meantime [20:03:51] polybuildr: try now? [20:03:51] legoktm: Worked. :D [20:03:51] woot [20:03:51] wanna +2 those patches then? [20:04:17] legoktm: uh oh [20:04:23] I think you gave me +2 on core too. [20:04:30] ummmmmm [20:04:37] wait what [20:04:44] you have a +2 button in mediawiki/core? [20:05:04] legoktm: I didn't check it that way, but I think so. Hold on, lemme confirm. [20:05:34] legoktm: okay no :P [20:05:37] phew [20:06:34] (03CR) 10Polybuildr: [C: 032] CharacterBeforePHPOpeningTagSniff: Support T_HASHBANG for HHVM >=3.5,<3.7 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/219394 (https://phabricator.wikimedia.org/T103119) (owner: 10Legoktm) [20:06:39] :D [20:06:45] legoktm: CR+2 and jenkins does the rest, right? [20:06:53] (03Merged) 10jenkins-bot: CharacterBeforePHPOpeningTagSniff: Support T_HASHBANG for HHVM >=3.5,<3.7 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/219394 (https://phabricator.wikimedia.org/T103119) (owner: 10Legoktm) [20:07:45] yep [20:07:55] legoktm: Isn't this where I get the "with great power comes great responsibility" lecture? [20:07:59] (03CR) 10Polybuildr: [C: 032] Update HISTORY.md and README.md for 0.3.0 release [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/219390 (owner: 10Legoktm) [20:08:01] (03Merged) 10jenkins-bot: Update HISTORY.md and README.md for 0.3.0 release [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/219390 (owner: 10Legoktm) [20:08:09] uhh, "don't be stupid" [20:08:17] woot [20:08:20] lemme push the tag [20:08:53] "You're therefore required to read this entire document and carefully review all the relevant links in it before using +2." sounds like such a serious thing. [20:14:54] and 0.3.0 is out :D [20:24:40] (03PS1) 10Krinkle: Create mwext-MobileFrontend-publish [integration/config] - 10https://gerrit.wikimedia.org/r/219449 (https://phabricator.wikimedia.org/T74794) [21:03:12] (03CR) 10Jdlrobson: [C: 031] "YESY YEYSYSYSY YYEEYSYS etc :D" [integration/config] - 10https://gerrit.wikimedia.org/r/219449 (https://phabricator.wikimedia.org/T74794) (owner: 10Krinkle) [21:06:43] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #675: FAILURE in 40 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/675/ [21:17:19] legoktm: https://gerrit.wikimedia.org/r/#/c/219470/, should we try to pass 0.2.0 first and then move to 0.3.0 or just do both in one go? [21:25:07] Node id: 35 added to jenkins [21:25:11] lets see what nodepool did [21:25:52] https://integration.wikimedia.org/ci/computer/ci-jessie-wikimedia-35/ !!! [21:28:46] 5Continuous-Integration-Isolation: Allow ssh from Jenkins master (gallium) to instances in contintcloud project - https://phabricator.wikimedia.org/T103184#1384012 (10hashar) 3NEW a:3hashar [21:30:40] 5Continuous-Integration-Isolation: Allow ssh from Jenkins master (gallium) to instances in contintcloud project - https://phabricator.wikimedia.org/T103184#1384038 (10hashar) 5Open>3Resolved Security rule has been applied :-} [21:31:58] 10Beta-Cluster: unable to connect to deployment-sql from deployment-video04 - https://phabricator.wikimedia.org/T43083#1384046 (10hashar) [21:42:48] 10Beta-Cluster, 6Labs, 10Wikimedia-Logstash, 5Patch-For-Review: Logstash on beta yields 500 due to NFS outage (can't open /data/project/logstash/.htpasswd) - https://phabricator.wikimedia.org/T102962#1384071 (10bd808) 5Open>3Resolved [21:42:51] 10Beta-Cluster, 6Labs: Things broken by betacluster suddenly being moved off NFS - https://phabricator.wikimedia.org/T102953#1384072 (10bd808) [21:55:00] hashar: neat-o :) [21:59:58] yeah [22:00:05] really need to get puppet to run in the image I am creating [22:00:26] i reached my goal for the week at least [22:00:59] and mobile pushing for the Android job -:-} ( https://phabricator.wikimedia.org/T62720 ) [22:01:03] hashar: that's awesome man, good work [22:01:10] oh [22:01:22] Andrew B. unlocked a shit ton of requirements [22:01:45] and I have read ton of doc, should be able to write some overview doc soonish [22:02:07] also [22:02:12] nodepool can boot docker containers :-] [22:02:36] well there is some stuff to convert a docker image into a format that is bootable on openstack [22:06:01] 6Release-Engineering, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, 6Multimedia, and 6 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1384154 (10faidon) Our CA for production is GlobalSign. It is one of the big (in terms of websites using it) and... [22:14:45] 10Continuous-Integration-Infrastructure: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1384173 (10Niedzielski) 3NEW [22:18:00] 5Continuous-Integration-Isolation: Write a big picture overview about what the CI isolation project is and a rough roadmap - https://phabricator.wikimedia.org/T102137#1384192 (10hashar) Some very basic Nodepool documentation at: https://wikitech.wikimedia.org/wiki/Nodepool [22:20:38] 10Continuous-Integration-Infrastructure: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1384198 (10hashar) [22:21:56] 10Continuous-Integration-Infrastructure, 10Ops-Access-Requests, 6operations: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1384200 (10Krenair) [22:22:49] 10Continuous-Integration-Infrastructure, 10Ops-Access-Requests, 6operations: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1384203 (10Krenair) [22:23:01] 10Continuous-Integration-Infrastructure, 10Ops-Access-Requests, 6operations: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1384204 (10hashar) p:5Triage>3Normal Thanks for the task! Lets wait for {T103191}. Once confirmed, you can be added via https:/... [22:24:23] AHHHHHHHHHHH [22:24:30] I can go to sleep :-} [22:24:32] https://integration.wikimedia.org/ci/computer/ci-jessie-wikimedia-38/ [22:24:39] created by nodepool and pooled in Jenkins \O/// [22:26:33] greg-g: ^^^^ [22:28:37] w00t! [22:35:05] and on Jessie which will please our ops :D [22:35:14] crash time to bed() have a good week-end [22:36:09] hashar: nice! [22:36:13] o/ [22:36:26] hashar: :) ! [22:36:49] it just took two years to reach that state hehe [22:51:36] 10Continuous-Integration-Infrastructure, 10Ops-Access-Requests, 6operations: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1384283 (10Dzahn) [23:05:50] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #162: FAILURE in 8 min 49 sec: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/162/ [23:22:22] 6Release-Engineering, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, 6Multimedia, and 6 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1384350 (10Tgr) [23:24:23] 6Release-Engineering, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, 6Multimedia, and 6 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1367794 (10Tgr) @faidon: fair enough (although https://www.ssllabs.com/ssltest/analyze.html?d=en.wikipedia.com cl... [23:31:39] 10Deployment-Systems, 6operations: tin doesn't have access to same memcached as terbium and app servers - https://phabricator.wikimedia.org/T103198#1384356 (10Mattflaschen) 3NEW [23:32:42] 6Release-Engineering, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, 6Multimedia, and 6 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1384366 (10BBlack) @Tgr that's because we don't speak the SSLv3 protocol anymore, because of the [[ https://en.wi... [23:33:06] 10Deployment-Systems, 6operations: tin doesn't have access to same memcached as terbium and app servers - https://phabricator.wikimedia.org/T103198#1384376 (10EBernhardson) tin config: ``` memcached: auto_eject_hosts: true distribution: ketama hash: md5...