[00:00:04] 10Continuous-Integration-Infrastructure, 6Labs: Designate should support split horizon resolution to yield private IP of instances behind a public DNS entry - https://phabricator.wikimedia.org/T95288#1286740 (10Andrew) I'm going to solve this by setting up a labs-specific recursor which will swizzle IPs the wa... [00:00:13] 10Continuous-Integration-Infrastructure, 6Labs: Designate should support split horizon resolution to yield private IP of instances behind a public DNS entry - https://phabricator.wikimedia.org/T95288#1286741 (10Andrew) [00:09:47] PROBLEM - Puppet staleness on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [43200.0] [00:11:39] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1286807 (10bd808) From the deploy tools side of this, it should be fairly simple to add a command to... [01:14:29] Hm.. deployment failed [01:14:29] 00:58:01 00:58:01 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n'] called with an empty host list. [01:14:36] For https://gerrit.wikimedia.org/r/#/c/211067/1/wmf-config/CommonSettings-labs.php [01:14:40] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/52901/console [01:15:40] Krinkle: that one is "normal" in beta. we don't have fanout rsync servers there any more [01:16:06] bd808: OK. Oh, so that's not the syncing itself. [01:16:24] no just the proxy update [01:16:31] Then where does hte actual syncing happen? [01:16:41] Somehow it didn't actually update, not sure what caused it [01:16:48] 00:58:09 Finished sync-apaches (duration: 00m 08s) [01:17:07] that is where the actual syncing happens [01:17:23] Hm. [01:18:14] Krinkle: why do oyu think it failed? that code isn't on the deployment-mediawiki0x's? [01:18:59] greg-g: Well, I know Special:Version doesn't list the extension [01:19:03] and Special:Interwiki is still 404 [01:19:09] maybe the code got synced but doesn't work. [01:19:49] :/ [01:21:21] [01:20 UTC] krinkle at deployment-mediawiki01.eqiad.wmflabs in /srv/mediawiki/wmf-config [01:21:21] $ grep Interwiki * [01:21:23] CommonSettings-labs.php:if ( $wmgEnableInterwiki ) { [01:21:23] CommonSettings-labs.php: require_once "$IP/extensions/Interwiki/Interwiki.php"; [01:21:25] CommonSettings-labs.php: $wgInterwikiViewOnly = true; [01:21:27] It did sync [01:22:47] Hm. deployment-bastion: eva.php --wiki enwiki [01:22:49] > var_dump($wmgEnableInterwiki); [01:22:49] bool(true) [01:22:51] So... [01:28:02] > var_dump( $wgInterwikiViewOnly); [01:28:03] bool(true) [01:28:03] > var_dump($wgMessagesDirs['Interwiki']); [01:28:04] string(63) "/mnt/srv/mediawiki-staging/php-master/extensions/Interwiki/i18n" [01:28:13] So the php file is actually being included as well [01:30:23] mw.loader.moduleRegistry['ext.interwiki.specialpage']> undefined [01:31:27] 10Continuous-Integration-Infrastructure, 10MediaWiki-RfCs: RFC: Extensions continuous integration - https://phabricator.wikimedia.org/T1350#1286863 (10Spage) [01:34:03] 10Deployment-Systems, 7HHVM: HHVM lock-ups - https://phabricator.wikimedia.org/T89912#1286874 (10BBlack) Looked at a live one on mw1169 today. I'm back to thinking this is all within StatCache somewhere, not the GenCountGuard stuff. In this example we've got many threads stuck on pthread mutex `0x2982578`.... [01:34:35] greg-g: Well, I don't know what's up, but somethign somewhere isn't working. [01:35:26] :) [01:35:36] I mean, suck, but... if you can't figure it out... [01:36:28] I'll file a bug for someone to figure out on Monday. [01:37:19] Krinkle: it's not a OMG FIX IT NOW thing right? [01:38:39] greg-g: Well, I assume that no deployments will work. Not just this config update. [01:38:52] So it's blocking any future change to beta cluster of any kind. [01:39:35] is it? or is just this code not working? [01:39:58] since the code is there, the deployment worked, no? [01:40:04] 10Beta-Cluster, 7Regression: Beta cluster deployments not working - https://phabricator.wikimedia.org/T99202#1286878 (10Krinkle) 3NEW [01:40:22] greg-g: Yeah, but all signals from the front-end indicate nothing was actually updated. [01:40:27] Maybe some kind of code cache. [01:40:47] It's not just http cache because I bypassed it client-side with Incognito and server-side with query parameters. [01:40:54] :/ [01:40:57] But yeah, it's possible the code is somehow not working properly. [01:41:05] I don't know. [02:25:15] Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce build #101: FAILURE in 3 min 14 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-linux-android-sauce/101/ [03:29:19] PROBLEM - Puppet staleness on deployment-redis01 is CRITICAL 100.00% of data above the critical threshold [43200.0] [04:08:14] 10Deployment-Systems, 7HHVM: HHVM lock-ups - https://phabricator.wikimedia.org/T89912#1287027 (10ori) [04:30:26] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #444: FAILURE in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/444/ [05:28:48] 6Release-Engineering, 10MediaWiki-Debug-Logging, 6Reading-Infrastructure-Team, 10Wikimedia-Logstash, 7HHVM: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#1287071 (10mmodell) not sure if this is related, but suddenly I'm seeing this on fatalmo... [05:34:51] Yippee, build fixed! [05:34:52] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #419: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/419/ [05:39:04] !log removed agent_catalog_run.lock on deployment-redis01. Inexplicably the file is dated Apr 24th? [05:43:31] 10Deployment-Systems, 10Staging, 6operations, 7Puppet: provider => trebuchet doesn't work until manual 'git deploy start' on deployment-server - https://phabricator.wikimedia.org/T92978#1287128 (10ArielGlenn) A run of salt my-deployment-server-here deploy.deployment_server_init will do the trick once I ge... [05:53:46] RECOVERY - Puppet staleness on deployment-redis01 is OK Less than 1.00% above the threshold [3600.0] [06:34:33] 6Release-Engineering, 10Hackathon-Lyon-2015: Hackathon Proposal: Wikimedia Site Requests Sprint - https://phabricator.wikimedia.org/T90468#1287186 (10Quiddity) [06:39:49] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [06:40:59] PROBLEM - Puppet failure on deployment-db2 is CRITICAL 50.00% of data above the critical threshold [0.0] [06:44:17] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [06:59:17] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.026 second response time [07:04:23] RECOVERY - Free space - all mounts on deployment-eventlogging02 is OK All targets OK [07:05:59] RECOVERY - Puppet failure on deployment-db2 is OK Less than 1.00% above the threshold [0.0] [08:15:54] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL 30.00% of data above the critical threshold [0.0] [08:16:18] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [08:17:00] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL 30.00% of data above the critical threshold [0.0] [08:21:16] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.019 second response time [08:40:56] RECOVERY - Puppet failure on deployment-sentry2 is OK Less than 1.00% above the threshold [0.0] [08:41:58] RECOVERY - Puppet failure on deployment-memc04 is OK Less than 1.00% above the threshold [0.0] [08:42:16] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [08:57:17] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.025 second response time [10:34:47] CFisch_WMDE1: grnh.se/gj5op4 [10:35:24] http://wikimediafoundation.org/wiki/Work_with_us [10:46:41] http://www.ratebeer.com/places/city/lyon/0/72/ [12:23:04] 10Beta-Cluster, 6Release-Engineering: Beta cluster "test.wikipedia" thinks it is "test.wikimedia" - https://phabricator.wikimedia.org/T99156#1287593 (10hashar) Related is {T97489} [12:28:08] 10Beta-Cluster: Interwiki extension not enabled on Beta cluster despite configuration change being applied - https://phabricator.wikimedia.org/T99202#1287597 (10hashar) [12:28:33] 10Beta-Cluster: Interwiki extension not enabled on Beta cluster despite configuration change being applied - https://phabricator.wikimedia.org/T99202#1286878 (10hashar) Seems the deployment are working fine but the Interwiki extension ends up not being loaded. There is other code that landed just fine. [13:00:31] -- [13:00:38] oops [13:00:40] sorry [13:08:02] ++ [14:03:34] Yippee, build fixed! [14:03:34] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » he,contintLabsSlave && UbuntuTrusty build #43: FIXED in 18 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=he,label=contintLabsSlave%20&&%20UbuntuTrusty/43/ [14:06:45] Yippee, build fixed! [14:06:45] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #249: FIXED in 44 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/249/ [14:13:02] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:18:48] PROBLEM - Puppet failure on deployment-salt is CRITICAL 60.00% of data above the critical threshold [0.0] [14:27:45] PROBLEM - Puppet failure on deployment-test is CRITICAL 20.00% of data above the critical threshold [0.0] [14:33:20] Yippee, build fixed! [14:33:20] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » es,contintLabsSlave && UbuntuTrusty build #44: FIXED in 18 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=es,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [14:43:00] RECOVERY - Puppet failure on deployment-memc04 is OK Less than 1.00% above the threshold [0.0] [14:43:45] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [14:53:39] Yippee, build fixed! [14:53:40] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » fi,contintLabsSlave && UbuntuTrusty build #44: FIXED in 39 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=fi,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [14:55:23] PROBLEM - Free space - all mounts on deployment-eventlogging02 is CRITICAL deployment-prep.deployment-eventlogging02.diskspace._var.byte_percentfree (<30.00%) [14:57:39] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [15:08:41] PROBLEM - Puppet failure on deployment-test is CRITICAL 30.00% of data above the critical threshold [0.0] [15:12:31] Yippee, build fixed! [15:12:32] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » tr,contintLabsSlave && UbuntuTrusty build #44: FIXED in 58 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=tr,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [15:31:08] Yippee, build fixed! [15:31:08] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » nl,contintLabsSlave && UbuntuTrusty build #44: FIXED in 1 hr 16 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=nl,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [15:32:25] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL 20.00% of data above the critical threshold [0.0] [15:34:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL 40.00% of data above the critical threshold [0.0] [15:41:33] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL 100.00% of data above the critical threshold [0.0] [15:43:39] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [15:50:53] Yippee, build fixed! [15:50:53] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » en,contintLabsSlave && UbuntuTrusty build #44: FIXED in 1 hr 36 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=en,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [15:52:27] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK Less than 1.00% above the threshold [0.0] [15:56:33] RECOVERY - Puppet failure on deployment-memc03 is OK Less than 1.00% above the threshold [0.0] [15:59:26] RECOVERY - Puppet failure on deployment-memc02 is OK Less than 1.00% above the threshold [0.0] [16:10:10] Yippee, build fixed! [16:10:11] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » fr,contintLabsSlave && UbuntuTrusty build #44: FIXED in 1 hr 55 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=fr,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [16:30:17] Yippee, build fixed! [16:30:18] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ru,contintLabsSlave && UbuntuTrusty build #44: FIXED in 2 hr 15 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ru,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [16:50:14] Yippee, build fixed! [16:50:15] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » id,contintLabsSlave && UbuntuTrusty build #44: FIXED in 2 hr 35 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=id,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [17:09:27] Yippee, build fixed! [17:09:27] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » om,contintLabsSlave && UbuntuTrusty build #44: FIXED in 2 hr 55 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=om,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [17:32:16] Yippee, build fixed! [17:32:17] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » krc,contintLabsSlave && UbuntuTrusty build #44: FIXED in 3 hr 17 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=krc,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [17:50:39] Yippee, build fixed! [17:50:40] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » nb,contintLabsSlave && UbuntuTrusty build #44: FIXED in 3 hr 36 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=nb,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [18:10:23] Yippee, build fixed! [18:10:24] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » kn,contintLabsSlave && UbuntuTrusty build #44: FIXED in 3 hr 56 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=kn,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [18:30:01] Yippee, build fixed! [18:30:02] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » sr,contintLabsSlave && UbuntuTrusty build #44: FIXED in 4 hr 15 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=sr,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [18:43:57] (03PS5) 10Krinkle: [WIP] Implement git-cache-update script [integration/jenkins] - 10https://gerrit.wikimedia.org/r/206074 (https://phabricator.wikimedia.org/T96687) [18:45:01] PROBLEM - Host deployment-apertium01 is DOWN: CRITICAL - Host Unreachable (10.68.16.79) [18:46:09] PROBLEM - Host deployment-bastion is DOWN: CRITICAL - Host Unreachable (10.68.16.58) [18:47:21] RECOVERY - Host deployment-apertium01 is UPING OK - Packet loss = 0%, RTA = 0.80 ms [18:50:40] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL 20.00% of data above the critical threshold [0.0] [18:50:40] Yippee, build fixed! [18:50:41] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » el,contintLabsSlave && UbuntuTrusty build #44: FIXED in 4 hr 36 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=el,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [18:52:24] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 33.33% of data above the critical threshold [0.0] [18:55:24] PROBLEM - Puppet failure on deployment-stream is CRITICAL 30.00% of data above the critical threshold [0.0] [18:56:08] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL 55.56% of data above the critical threshold [0.0] [18:57:16] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL 66.67% of data above the critical threshold [0.0] [19:00:01] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL 30.00% of data above the critical threshold [0.0] [19:02:16] Project beta-scap-eqiad build #53009: FAILURE in 17 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/53009/ [19:09:21] Yippee, build fixed! [19:09:22] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ro,contintLabsSlave && UbuntuTrusty build #44: FIXED in 4 hr 54 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ro,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [19:11:07] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL 55.56% of data above the critical threshold [0.0] [19:29:15] Yippee, build fixed! [19:29:16] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ar,contintLabsSlave && UbuntuTrusty build #44: FIXED in 5 hr 14 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ar,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [19:31:52] (03CR) 10Hashar: "I am not a fan of PHP anymore, some comments regardless :-]" (036 comments) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/206074 (https://phabricator.wikimedia.org/T96687) (owner: 10Krinkle) [19:34:09] (03Abandoned) 10Hashar: Beta: Add wikis for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/206389 (https://phabricator.wikimedia.org/T90683) (owner: 10Hashar) [19:47:58] Yippee, build fixed! [19:47:58] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ja,contintLabsSlave && UbuntuTrusty build #44: FIXED in 5 hr 33 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ja,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [19:48:28] don't yell at me, I am not here. [19:48:32] working on self review tonight [20:06:53] Yippee, build fixed! [20:06:54] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » et,contintLabsSlave && UbuntuTrusty build #44: FIXED in 5 hr 52 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=et,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [20:25:03] Yippee, build fixed! [20:25:04] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » pt,contintLabsSlave && UbuntuTrusty build #44: FIXED in 6 hr 10 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=pt,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [20:41:08] RECOVERY - Host deployment-bastion is UPING OK - Packet loss = 0%, RTA = 1.38 ms [20:48:21] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: No route to host [20:49:41] PROBLEM - Host deployment-cache-mobile03 is DOWN: CRITICAL - Host Unreachable (10.68.16.13) [20:55:41] PROBLEM - Host deployment-db2 is DOWN: CRITICAL - Host Unreachable (10.68.17.94) [20:56:05] RECOVERY - Puppet failure on deployment-mediawiki01 is OK Less than 1.00% above the threshold [0.0] [20:57:22] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [21:00:36] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [21:00:44] RECOVERY - Host deployment-db2 is UPING OK - Packet loss = 0%, RTA = 0.91 ms [21:01:08] RECOVERY - Puppet failure on deployment-mediawiki02 is OK Less than 1.00% above the threshold [0.0] [21:02:18] RECOVERY - Puppet failure on deployment-restbase01 is OK Less than 1.00% above the threshold [0.0] [21:05:24] RECOVERY - Puppet failure on deployment-stream is OK Less than 1.00% above the threshold [0.0] [21:10:01] RECOVERY - Puppet failure on deployment-mediawiki03 is OK Less than 1.00% above the threshold [0.0] [21:12:21] PROBLEM - Host deployment-logstash1 is DOWN: CRITICAL - Host Unreachable (10.68.16.134) [21:18:11] PROBLEM - Host deployment-memc02 is DOWN: CRITICAL - Host Unreachable (10.68.16.14) [21:20:14] Yippee, build fixed! [21:20:14] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » lb,contintLabsSlave && UbuntuTrusty build #44: FIXED in 7 hr 5 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=lb,label=contintLabsSlave%20&&%20UbuntuTrusty/44/ [22:39:51] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #672: FAILURE in 28 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/672/ [22:47:21] (03PS6) 10Krinkle: Implement git-cache-update script [integration/jenkins] - 10https://gerrit.wikimedia.org/r/206074 (https://phabricator.wikimedia.org/T96687) [22:47:54] PROBLEM - Host integration-slave-precise-1014 is DOWN: CRITICAL - Host Unreachable (10.68.18.38) [22:48:18] (03CR) 10Krinkle: Implement git-cache-update script (035 comments) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/206074 (https://phabricator.wikimedia.org/T96687) (owner: 10Krinkle) [22:48:58] (03CR) 10jenkins-bot: [V: 04-1] Implement git-cache-update script [integration/jenkins] - 10https://gerrit.wikimedia.org/r/206074 (https://phabricator.wikimedia.org/T96687) (owner: 10Krinkle) [22:52:43] RECOVERY - Host integration-slave-precise-1014 is UPING OK - Packet loss = 0%, RTA = 0.71 ms [22:54:17] (03CR) 10Krinkle: "recheck" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/206074 (https://phabricator.wikimedia.org/T96687) (owner: 10Krinkle) [22:56:26] PROBLEM - Host integration-slave-trusty-1016 is DOWN: CRITICAL - Host Unreachable (10.68.18.34) [23:01:21] RECOVERY - Host integration-slave-trusty-1016 is UPING OK - Packet loss = 0%, RTA = 0.83 ms [23:06:36] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Regression: Majority of Jenkins slaves exceed acceptable clockdrift (more than 60 seconds) - https://phabricator.wikimedia.org/T99304#1289574 (10Krinkle) 3NEW [23:20:20] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Regression: Majority of Jenkins slaves exceed acceptable clockdrift (more than 60 seconds) - https://phabricator.wikimedia.org/T99304#1289616 (10Andrew) This is probably due to the security patch that I'm applying now (via a suspend/resume). Yo... [23:31:21] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL 100.00% of data above the critical threshold [0.0] [23:31:25] PROBLEM - Puppet staleness on deployment-urldownloader is CRITICAL 100.00% of data above the critical threshold [43200.0] [23:31:45] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0] [23:37:51] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Regression: Majority of Jenkins slaves exceed acceptable clockdrift (more than 60 seconds) - https://phabricator.wikimedia.org/T99304#1289672 (10Krinkle) 5Open>3Resolved a:3Krinkle Thanks. [23:38:22] (03CR) 10Krinkle: "Failure is false negative, T99304." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/206074 (https://phabricator.wikimedia.org/T96687) (owner: 10Krinkle) [23:38:27] (03CR) 10Krinkle: "recheck" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/206074 (https://phabricator.wikimedia.org/T96687) (owner: 10Krinkle) [23:46:33] 10Beta-Cluster, 10MediaWiki-JobQueue: Job queue can't insert jobs on Beta Cluster - https://phabricator.wikimedia.org/T99311#1289709 (10Mattflaschen) 3NEW [23:57:35] RECOVERY - Host deployment-cache-mobile03 is UPING OK - Packet loss = 0%, RTA = 0.68 ms [23:58:21] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 31027 bytes in 0.478 second response time