[00:17:45] Project selenium-Flow » chrome,beta,Linux,BrowserTests build #472: 04FAILURE in 1 min 44 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/472/ [00:17:55] Project selenium-Flow » firefox,beta,Linux,BrowserTests build #472: 04FAILURE in 1 min 54 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/472/ [00:53:08] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<44.44%) [01:44:31] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:53] ^ just slow [01:49:24] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 54767 bytes in 1.168 second response time [02:50:46] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3499665 (10Dzahn) [03:28:58] 10Release-Engineering-Team, 10MediaWiki-JobRunner, 10Operations, 10monitoring, 10Wikimedia-Incident: Collect error logs from jobchron/jobrunner services in Logstash - https://phabricator.wikimedia.org/T172479#3499719 (10Krinkle) [03:40:36] 10Deployment-Systems, 10Release-Engineering-Team: Add jobrunners to Scap canary process - https://phabricator.wikimedia.org/T172480#3499737 (10Krinkle) [03:41:44] 10Release-Engineering-Team, 10MediaWiki-JobRunner, 10Operations, 10monitoring, 10Wikimedia-Incident: Collect error logs from jobchron/jobrunner services in Logstash - https://phabricator.wikimedia.org/T172479#3499754 (10Krinkle) [04:24:11] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3499773 (10mmodell) [04:29:34] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T170631#3499794 (10mmodell) [04:30:00] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3499795 (10mmodell) [05:47:23] (03Draft2) 10Florianschmidtwelzow: Archive SemanticUpdateOnPurge [integration/config] - 10https://gerrit.wikimedia.org/r/370148 (https://phabricator.wikimedia.org/T172294) [06:41:10] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:53:08] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:23:16] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3499923 (10Dzahn) [07:27:59] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator: reinstall iridium (phabricator) as phab1001 with jessie - https://phabricator.wikimedia.org/T152129#3499927 (10mmodell) 05Open>03Resolved [07:28:31] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3499928 (10Dzahn) [07:30:17] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3273664 (10Dzahn) T152129 has been resolved [07:30:40] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Patch-For-Review: Switch phabricator production to codfw - https://phabricator.wikimedia.org/T164810#3499934 (10mmodell) [07:31:17] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Provide cross-dc redundancy (active-active or active-passive) to all important misc services - https://phabricator.wikimedia.org/T156937#3499936 (10mmodell) [07:37:45] 10Scap, 10Phabricator: scap should provide a way to skip symlink-swapping in promote - https://phabricator.wikimedia.org/T172486#3499941 (10mmodell) [07:41:14] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [07:45:33] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3499968 (10Dzahn) [07:45:54] 10Release-Engineering-Team, 10Operations, 10Phabricator, 10hardware-requests, 10ops-eqiad: replacement hardware for iridium (phabricator) - https://phabricator.wikimedia.org/T156970#3499972 (10Dzahn) [07:45:59] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Patch-For-Review: setup/install phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T163938#3273670 (10Dzahn) 05Open>03Resolved [08:41:12] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [08:44:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Joe: CI for operations/puppet is taking too long - https://phabricator.wikimedia.org/T166888#3310890 (10Joe) I did rewrite the Rakefile according to @faidon's suggestions, did tweak the... [08:45:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Joe: CI for operations/puppet is taking too long - https://phabricator.wikimedia.org/T166888#3500147 (10Joe) 05Open>03Resolved [09:01:11] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [11:20:31] oh wow https://phabricator-new.wikimedia.org/diffusion/ is connected to codfw :) [11:20:38] twentyafterfour mutante ^^ :) [11:23:52] 10Scap, 10Analytics, 10EventBus, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3500541 (10elukey) [11:24:43] 10Scap, 10Analytics, 10EventBus, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3467218 (10elukey) @Joe what are the available options? Shall we migrate to use deploy-service for eventlogging or is there a way t... [11:40:31] 10Gitblit-Deprecate, 10Epic, 10MW-1.30-release-notes (WMF-deploy-2017-07-18_(1.30.0-wmf.10)), 10Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#3500582 (10Aklapper) Well, https://github.com/search?p=2&q=org%3Awikimedia+%22git.wikimedia.org%22&t... [12:08:51] 10Release-Engineering-Team (Kanban), 10User-greg, 10Wikimedia-Incident: Identify "first responders" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#3500658 (10Aklapper) [13:07:34] sigh still running into that publickey error [13:18:07] strangely it gets fixed doing ssh-add each time [13:18:17] I though it would stay forever :| [13:18:38] TabbyCat: Do you keep rebooting your machine? [13:19:03] Reedy: what do you mean, if I shut down my computer after I finish my work? Then yes [13:19:14] that's your problem [13:19:20] ssh-add is persistent for as long as the OS is running [13:19:27] ah [13:19:28] Shutdown or reboot will remove it [13:19:53] :D [13:19:55] thanks [13:22:07] otoh I followed https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_.28recommended.29 and does not seem to work [13:28:36] TabbyCat why not do what i suggested yesturday? :) [13:29:38] paladox: I did! :) But that's not the host I want to access :) [13:29:38] TabbyCat you need to do ssh [13:29:49] TabbyCat you add it for each host [13:29:52] paladox: an example please? [13:30:08] ie, for deployment-tin.deployment-prep.equiad.wmflabs [13:30:23] soemthing like [13:30:24] https://phabricator.wikimedia.org/P5858 [13:30:28] what's your username? [13:30:57] https://phabricator.wikimedia.org/P5850#31999 [13:31:02] let's use that ;) [13:31:32] https://phabricator.wikimedia.org/P5859 [13:31:36] TabbyCat ^^ [13:31:41] replace the username with yours [13:31:46] then do [13:31:51] ssh deployment-tin [13:31:59] okay but shell is maurelio [13:32:01] I'll do that [13:32:03] ok [13:32:04] * TabbyCat tests [13:33:27] paladox: it works ;D [13:33:31] :) [13:33:34] *hugs* [13:33:46] that should now work regardless if you reboot :) [13:33:59] you need to do that for each host you want to connect to though [13:34:52] well, for tools I just ssh maurelio@tools-login.etc and it works just fine [13:35:06] and then become [13:38:23] ok [14:10:18] hmm https://logstash-beta.wmflabs.org/app/kibana#/doc/logstash-*/logstash-2017.08.04/mediawiki?id=AV2tgg__hvX_CkfKqjK5&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-1h,mode:quick,to:now)) [14:10:48] what I noticed is that I'm being constantly logged-in/logged-out there [14:12:11] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Services (doing), 10User-mobrovac, 10Wikimedia-Hackathon-2015: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3501073 (10dbarratt) I created an [[ https://github.com/docker-library/official-images/issues/3271 | issue ]... [14:22:26] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:31] 10Beta-Cluster-Infrastructure, 10User-MarcoAurelio, 10Wikimedia-log-errors: Have to log-in each time I switch to a different page or perform an action in the Beta Cluster - https://phabricator.wikimedia.org/T172526#3501117 (10MarcoAurelio) [14:32:22] RECOVERY - Parsoid on deployment-parsoid09 is OK: HTTP OK: HTTP/1.1 200 OK - 1051 bytes in 3.681 second response time [14:33:46] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #475: 04FAILURE in 1 min 46 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/475/ [14:38:25] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: HTTP CRITICAL - No data received from host [14:43:22] RECOVERY - Parsoid on deployment-parsoid09 is OK: HTTP OK: HTTP/1.1 200 OK - 1051 bytes in 2.808 second response time [15:08:04] 10Scap, 10Analytics, 10EventBus, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3501376 (10elukey) So the following class is responsible to add the necessary credentials to the `deploy-service`: ``` # === Class... [15:11:38] 10Scap, 10Analytics, 10EventBus, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3501418 (10Ottomata) I am not strongly opposed to using deploy-service. But, this means that when we switch eventlogging analytics... [15:12:53] PROBLEM - Puppet errors on deployment-ms-fe02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:14:01] 10Scap, 10Analytics, 10EventBus, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3501423 (10Ottomata) For example, we'd like to restrict kafka producers and consumers to specific topics by adding ACLs for TLS pri... [15:17:24] (03PS1) 10Thcipriani: Docker: operations-puppet-docker voting/only job [integration/config] - 10https://gerrit.wikimedia.org/r/370222 [15:18:23] (03PS2) 10Thcipriani: Docker: operations-puppet-docker voting/only job [integration/config] - 10https://gerrit.wikimedia.org/r/370222 [15:19:05] (03CR) 10Thcipriani: [C: 032] Docker: operations-puppet-docker voting/only job [integration/config] - 10https://gerrit.wikimedia.org/r/370222 (owner: 10Thcipriani) [15:20:14] (03Merged) 10jenkins-bot: Docker: operations-puppet-docker voting/only job [integration/config] - 10https://gerrit.wikimedia.org/r/370222 (owner: 10Thcipriani) [15:21:36] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/370222/2 [15:21:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:07:31] 10Scap, 10Analytics, 10EventBus, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3467218 (10thcipriani) >>! In T171506#3501376, @elukey wrote: > So the following class is responsible to add the necessary credenti... [17:08:52] 10Scap, 10Global-Collaboration, 10MediaWiki-extensions-LocalisationUpdate, 10User-Nikerabbit: Alert when l10update fails - https://phabricator.wikimedia.org/T171925#3501698 (10Nikerabbit) [17:11:43] 10Scap, 10Analytics, 10EventBus, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3467218 (10mobrovac) >>! In T171506#3501418, @Ottomata wrote: > But, this means that when we switch eventlogging analytics use to d... [17:33:00] (03PS1) 10Umherirrender: Fix phpunit test on windows [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/370229 [17:37:16] 10Scap, 10Analytics, 10EventBus, 10User-Elukey: eventlogging-service-eventbus scap deployments should depool/pool during deployment - https://phabricator.wikimedia.org/T171506#3501792 (10Ottomata) AH! Interesting! If so, then totally fine to deploy with deploy-service, and run with user eventlogging. Us... [17:41:11] Hmm. Should we move pywikibot out of the Wikimedia grouping on GitHub? https://travis-ci.org/wikimedia/pywikibot-core/builds/261050113 blocked merges via Travis for four hours(!) because the huge number of tests they run… [17:41:40] you just mentioned two things we don't know/care much about ;) [17:41:57] greg-g: But other people who hang in here do. :-) [17:42:17] true :) hello others who care ^ ;) [17:42:32] but, I'd wager "yes"? [17:43:22] It'd probably break Chad's magic gerrit->GitHub repo sync tool, though. [17:43:25] Meh. [17:43:40] (03PS1) 10Umherirrender: Fix Undefined offset in FunctionCommentSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/370231 (https://phabricator.wikimedia.org/T171776) [17:46:45] oh... [17:46:49] hrmmm [17:48:02] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Services (doing), 10User-mobrovac, 10Wikimedia-Hackathon-2015: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3501836 (10dbarratt) I created a very basic mediawiki image: https://hub.docker.com/r/davidbarratt/mediawiki... [17:49:55] (03PS1) 10Umherirrender: Enforce "short" type definitions on @param in comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/370232 (https://phabricator.wikimedia.org/T145162) [17:57:17] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Services (doing), 10User-mobrovac, 10Wikimedia-Hackathon-2015: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3501860 (10Addshore) @dbarratt where is the code for said image? I dont see it linked above or on docker or... [17:57:55] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Services (doing), 10User-mobrovac, 10Wikimedia-Hackathon-2015: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3501861 (10Addshore) Scratch that.... https://github.com/davidbarratt/mediawiki-docker [18:03:57] 10MediaWiki-Codesniffer, 10Patch-For-Review: Undefined offset: 1 on codesniffer FunctionCommentSniff - https://phabricator.wikimedia.org/T171776#3501864 (10Umherirrender) p:05Triage>03Normal a:03Umherirrender [18:06:44] 10MediaWiki-Codesniffer: Double spacing - https://phabricator.wikimedia.org/T172551#3501871 (10Reedy) [18:06:52] 10MediaWiki-Codesniffer: Double spacing - https://phabricator.wikimedia.org/T172551#3501883 (10Reedy) p:05Triage>03Lowest [18:33:21] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.30.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T168053#3501941 (10MaxSem) [18:37:25] 10MediaWiki-Releasing, 10MediaWiki-Containers, 10Services (doing), 10User-mobrovac, 10Wikimedia-Hackathon-2015: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#3501979 (10dbarratt) I was going to create a default `LocalSettings.php` file that used environment variable... [18:45:55] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:46:11] greg-g: Hi [18:49:04] greg-g: What's the way forward? [18:56:19] d3r1ck: context? (also, I'm lagging right now, just fyi) [19:10:53] 10MediaWiki-Codesniffer, 10Patch-For-Review: Provide Codesniffer rules to enforce "short" type definitions (int/bool, not integer/boolean) - https://phabricator.wikimedia.org/T145162#3502032 (10Umherirrender) Have seen "@return integer|bool", which is not covered - also not from my patch set for @param [19:30:36] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3502076 (10Mattflaschen-WMF) [19:38:41] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3502093 (10Mattflaschen-WMF) [19:39:54] greg-g, tgr|away, noticeable bug making it a pain to use Beta: T172560 [19:39:54] T172560: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560 [20:03:51] (03Abandoned) 10Umherirrender: Stop reference of string $content as an array [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/283384 (https://phabricator.wikimedia.org/T127572) (owner: 10Aashaka) [20:04:27] 10MediaWiki-Codesniffer, 10Patch-For-Review: "Uninitialized string offset: 72" errors from SpaceBeforeSingleLineCommentSniff.php - https://phabricator.wikimedia.org/T127572#3502124 (10Umherirrender) 05Open>03Invalid Handled by T114633, but not a duplicate [20:07:15] (03Abandoned) 10Umherirrender: Improve invalid whitespaces in comment detection. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/276466 (https://phabricator.wikimedia.org/T129474) (owner: 10Schenkerx) [20:18:01] 10MediaWiki-Codesniffer: Fix SpaceyParenthesisSniff can't fix not single space after opening parenthesis and before closing parenthesis - https://phabricator.wikimedia.org/T129231#3502151 (10Umherirrender) 05Open>03Resolved [20:42:58] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #476: 04FAILURE in 1 min 55 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/476/ [21:14:49] matt_flaschen: :/ [21:17:50] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3502365 (10greg) [21:17:52] 10Beta-Cluster-Infrastructure, 10User-MarcoAurelio, 10Wikimedia-log-errors: Have to log-in each time I switch to a different page or perform an action in the Beta Cluster - https://phabricator.wikimedia.org/T172526#3502367 (10greg) [21:30:36] 10Release-Engineering-Team (Watching / External), 10MediaWiki-JobRunner, 10MediaWiki-Platform-Team, 10Operations, and 2 others: Collect error logs from jobchron/jobrunner services in Logstash - https://phabricator.wikimedia.org/T172479#3502395 (10greg) Adding #mediawiki-platform-team as @aaron is maintaine... [21:34:04] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:47] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:36:47] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:38:43] 10Release-Engineering-Team (Kanban), 10User-greg: Setup 15 MW-Vagrant USB sticks for Wikimania 2017 Hackathon - https://phabricator.wikimedia.org/T172334#3502421 (10greg) Boy this task is bothersome. Writing 3gb to usb sticks really makes my laptop unresponsive for normal things. About half-way through. Will d... [22:23:41] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3502606 (10Mattflaschen-WMF) Fixing the fatal @MarcoAurelio linked to. Don't know if that will fix this issue, though. [22:55:59] twentyafterfour: i'll change the UID of the "phd" user on phab2001, so that it matches the UID on phab1001. then we won't have permission issues with rsync anymore [22:56:13] mutante: good idea [22:56:18] since there are no running processes owned by phd and that particular UID is also not taken [22:56:27] i will change it, then run find with -exec [22:56:31] reboot the box [22:57:01] i have done it before, you had stuff like cron had to be restarted, but can be done [22:57:19] it will be nicer for a cluster [22:57:36] if you wonder "but normally rsync can handle this and do it by username" [22:58:11] then the answer is "yea, normally, but [22:58:12] "rsync forces --numeric-ids when fetching content from an rsyncd configured with use chroot = yes" [22:58:27] also https://phabricator.wikimedia.org/T79786#1831969 [22:58:38] which is the same problem ,just mw user [22:58:54] in our case we just have to fix 1 box though and not all appservers ... [23:47:17] twentyafterfour: do you recall editing /etc/hosts on phab2001 to test something? [23:48:38] the rsync stuff wasnt working and it was behaving really weird.. like if you use "ping phab1001 or telnet1001" you got the old IP, but host/dig showed the correct new IP.. then i found we have them in /etc/hosts. remove or update .. hmm [23:49:06] removes them