[00:18:04] 3Beta-Cluster: Thumbnail 404s get cached - https://phabricator.wikimedia.org/T69056#799964 (10Tgr) I think this is not beta-related but a bug (or quirk) of how thumb.php works. What happens is basically that thumb.php does not do remote file fetches (T27958); it returns a 404 instead, and that 404 gets cached i... [00:23:28] greg-g: I think the scap patch to fix the l10nupdate process is ready to go out, but I didn't have time to babysit it rolling out today. :( If you can find a victim^Wvolunteer to do that, +2. [02:04:15] 3Continuous-Integration: [OPS] hhvm 3.3.0-20140925+wmf3 has some annoying build dependency - https://phabricator.wikimedia.org/T73413#800218 (10Liuxinyu970226) [03:23:20] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #263: FAILURE in 1 min 51 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/263/ [03:37:10] Yippee, build fixed! [03:37:10] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #110: FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/110/ [03:42:36] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [04:10:53] (03CR) 10Mattflaschen: [C: 031] Limit browser tests history to 31 days [integration/config] - 10https://gerrit.wikimedia.org/r/176649 (owner: 10Hashar) [04:30:41] 3Release-Engineering, MediaWiki-Core-Team: Make ::mediawiki::syslog and ::mediawiki::php logging destinations configurable via hiera - https://phabricator.wikimedia.org/T1295#800356 (10bd808) p:5Triage>3High [04:36:04] Yippee, build fixed! [04:36:05] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #303: FIXED in 2 min 43 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/303/ [04:52:40] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #389: FAILURE in 1 hr 0 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/389/ [05:22:15] 3Beta-Cluster: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#800385 (10Krinkle) There's at least one consumer of graphite.wmflabs.org (Nagf; https://github.com/wikimedia/nagf; https://tools.wmflabs.org/nagf/) that assumes the root level to be Nova pro... [05:26:10] !log integration-slave1001 has been down since the failed reboot on 28 November 2014. Still unreachable over ssh and no Jenkins slave agent. [05:26:15] Logged the message, Master [05:32:39] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [05:44:56] Yippee, build fixed! [05:44:57] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #257: FIXED in 1 min 20 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/257/ [05:58:36] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:16:14] Yippee, build fixed! [06:16:14] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #293: FIXED in 4 min 52 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/293/ [06:38:46] Yippee, build fixed! [06:38:46] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #261: FIXED in 1 min 45 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/261/ [07:33:40] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [08:04:37] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:56:54] 3Phabricator, Research-and-Data, MediaWiki-Core-Team, Engineering-Community, WMF-Design, Wikidata, Mobile-Apps, Multimedia, Services, Zero, Language-Engineering, Release-Engineering, Editing, Core-Features, Parsoid-Team, Mobile-Web, Scrum-of-Scrums: Create team projects for all teams participating in scrum of scru... [08:58:48] Yippee, build fixed! [08:58:49] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #199: FIXED in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/199/ [09:01:05] aharoni: I am in the hangout [09:06:54] Yippee, build fixed! [09:06:54] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #319: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/319/ [09:40:32] Yippee, build fixed! [09:40:33] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #321: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/321/ [10:18:21] Yippee, build fixed! [10:18:22] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #193: FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/193/ [10:23:33] !log restarted parsoid on deployment-parsoid05 [10:23:36] Logged the message, Master [10:24:07] Yippee, build fixed! [10:24:07] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #320: FIXED in 5 min 27 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/320/ [10:28:06] (03CR) 10Zfilipin: [C: 031] Try local gem cache in bundle job [integration/config] - 10https://gerrit.wikimedia.org/r/176818 (owner: 10Dduvall) [10:44:38] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [10:52:55] (03CR) 10Hashar: [C: 04-1] "bundle will end up stalled at some version and never upgrade, we could use a minimum version requirement to ease upgrading." (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/176818 (owner: 10Dduvall) [11:11:37] 3Continuous-Integration, OOjs-UI: PHP docs should be auto-generated - https://phabricator.wikimedia.org/T74454#750834 (10hashar) We can do it for branches, but not for tags currently :( [11:12:31] Yippee, build fixed! [11:12:32] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #265: FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/265/ [11:20:57] Yippee, build fixed! [11:20:58] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #315: FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/315/ [11:23:36] Yippee, build fixed! [11:23:37] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #339: FIXED in 1 hr 17 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/339/ [11:24:02] 3monitoring, Beta-Cluster: Shinken warnings about free space on beta cluster Varnish instances - https://phabricator.wikimedia.org/T76417#801067 (10hashar) [11:24:38] 3monitoring, Beta-Cluster: Shinken warnings about free space on beta cluster Varnish instances - https://phabricator.wikimedia.org/T76417#799841 (10hashar) [11:26:24] 3monitoring, Beta-Cluster: Shinken warnings about free space on beta cluster Varnish instances - https://phabricator.wikimedia.org/T76417#801071 (10hashar) When discussing with Brandon about an unrelated Varnish bug, he instructed me to delete the files entirely and restart Varnish (tracked by T76091 which I mad... [11:26:54] Project browsertests-VisualEditor-production-linux-firefox-sauce build #31: ABORTED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-production-linux-firefox-sauce/31/ [11:27:32] !log deleting /srv/vdb/varnish* files on all varnish instances ( https://phabricator.wikimedia.org/T76091 ) [11:27:33] Logged the message, Master [11:33:19] 3monitoring, Beta-Cluster: Shinken warnings about free space on beta cluster Varnish instances - https://phabricator.wikimedia.org/T76417#801093 (10hashar) [11:33:20] 3Beta-Cluster: Delete /srv/vdb/varnish* files on all Varnish instances - https://phabricator.wikimedia.org/T76091#801090 (10hashar) 5Open>3Resolved a:3hashar I have deleted the files using: `service varnish stop && rm -v *; service varnish start` On instances: * deployment-cache-mobile03 * deployment-cac... [11:44:43] RECOVERY - Free space - all mounts on deployment-sca-cache01 is OK: OK: All targets OK [11:46:01] 3Beta-Cluster: Local cherry-picks missing on deployment-salt - https://phabricator.wikimedia.org/T75947#801104 (10hashar) 5Open>3Resolved a:3hashar The local hacks / cherry picks have been restored so this task is complete. We have another task to get rid of the local hacks: {T76392} [12:02:53] lubaochuan: how do I join the event? [12:03:09] hold on. [12:03:11] I can see you at youtube, but there is no link to join the event [12:07:52] lets use the event you created. [12:08:41] lubaochuan: ok, see you there [12:09:40] https://plus.google.com/hangouts/_/wikimedia.org/zfilipin [12:17:30] lubaochuan: https://leanpub.com/cucumber_and_cheese [14:53:32] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:55:21] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:15:52] Project browsertests-VisualEditor-production-linux-firefox-sauce build #32: STILL FAILING in 3 hr 42 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-production-linux-firefox-sauce/32/ [15:18:34] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:20:18] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:36:56] 3Continuous-Integration: zuul-cloner does not support Ref events - https://phabricator.wikimedia.org/T76003#801782 (10hashar) Needed another patch to upstream: https://review.openstack.org/#/c/138433/ [17:18:48] 3Quality-Assurance: Run Selenium tests in parallel - https://phabricator.wikimedia.org/T57867#801923 (10zeljkofilipin) p:5Normal>3Low [17:20:08] 3Quality-Assurance: investigate using ChromeDriver on Android - https://phabricator.wikimedia.org/T74641#801928 (10zeljkofilipin) I think [[ http://appium.io/ | Appium ]] is the way to go, it is supported by Sauce Labs. [17:20:28] 3Quality-Assurance: investigate using ChromeDriver on Android - https://phabricator.wikimedia.org/T74641#801932 (10zeljkofilipin) p:5Triage>3Low [17:21:03] 3Quality-Assurance: Jenkins tests shouldn't go red when it's not its fault - https://phabricator.wikimedia.org/T74722#801937 (10zeljkofilipin) [17:21:16] 3Quality-Assurance: Jenkins tests shouldn't go red when it's not its fault - https://phabricator.wikimedia.org/T74722#801938 (10zeljkofilipin) [17:48:51] 3VisualEditor, Beta-Cluster: VisualEditor not updating in beta labs - https://phabricator.wikimedia.org/T76097#789475 (10zeljkofilipin) [17:50:54] 3VisualEditor, Beta-Cluster: VisualEditor not updating in beta labs - https://phabricator.wikimedia.org/T76097#802032 (10hashar) Seems the script python I wrote to update the VisualEditor repo is no more working / choking on something. I haven't noticed this task until it has be pointed to me during the Releng... [17:51:47] 3Scrum-of-Scrums, Beta-Cluster: Create labs ldap user and group "apache" to be used for NFS4 access in deployment-prep project - https://phabricator.wikimedia.org/T76086#802034 (10Andrew) I just added users Apache with uid/gid 48/48. I'm not 100% positive this won't cause conflicts on some systems due to the un... [18:23:36] Yippee, build fixed! [18:23:36] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #264: FIXED in 1 min 42 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/264/ [18:37:09] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #111: FAILURE in 35 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/111/ [18:46:39] 3Wikimedia-Git-or-Gerrit, Continuous-Integration: labs-tools-grrrit-yamllint job broken? - https://phabricator.wikimedia.org/T76508#802122 (10Legoktm) [18:50:25] 3Wikimedia-Git-or-Gerrit, Continuous-Integration: labs-tools-grrrit-yamllint job broken? - https://phabricator.wikimedia.org/T76508#802134 (10AzaToth) [19:54:42] Yippee, build fixed! [19:54:42] Project beta-scap-eqiad build #32242: FIXED in 4 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32242/ [19:56:11] Yippee, build fixed! [19:56:11] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #390: FIXED in 1 hr 6 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/390/ [20:24:30] Yippee, build fixed! [20:24:30] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #220: FIXED in 16 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/220/ [20:32:51] 3VisualEditor, Beta-Cluster: VisualEditor not updating in beta labs - https://phabricator.wikimedia.org/T76097#802330 (10hashar) 5Open>3Resolved a:3hashar Seems there was some transient error that prevented Gerrit to merge the bot change automatically. Apparently due to some merge conflict. I am not sure... [20:48:00] 3Continuous-Integration, Beta-Cluster, Release-Engineering: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#802372 (10Jdforrester-WMF) [21:02:44] kaldari|2: I'm trying to figure out something in the browser tests and I'm stumped. Why does this page have a redlink http://en.m.wikipedia.beta.wmflabs.org/wiki/0.9942907553443305alklkh but this one does not? http://en.m.wikipedia.beta.wmflabs.org/wiki/NewPage1417553647 [21:03:30] oops ww [21:46:36] Yippee, build fixed! [21:46:36] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #202: FIXED in 15 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/202/ [22:49:45] (03CR) 10Krinkle: "bug 12 ." [integration/config] - 10https://gerrit.wikimedia.org/r/174731 (owner: 10Krinkle) [23:09:18] (03CR) 10Krinkle: "Foo [bug 12]" [integration/config] - 10https://gerrit.wikimedia.org/r/174731 (owner: 10Krinkle) [23:15:49] Project beta-scap-eqiad build #32262: FAILURE in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32262/ [23:18:49] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:24:46] PROBLEM - Free space - all mounts on deployment-mediawiki01 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki01.diskspace._var.byte_percentfree.value (<30.00%) [23:25:12] PROBLEM - Free space - all mounts on deployment-mediawiki02 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki02.diskspace._var.byte_percentfree.value (<44.44%) [23:25:42] I'm seeing some weird Ruby gems fail here, https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/38/console [23:28:37] awight: dnsimple experienced a ddos yesterday that took down rubygems [23:28:50] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:30:39] and heroku and a few other sites [23:31:17] marxarel_: ah ok then! [23:31:48] (mebbe we should be mirroring that stuff locally for security reasons, anyway) [23:35:16] !log /var full on deployment-mediawiki01 [23:35:20] Logged the message, Master [23:39:00] !log Cause of full disk on deployment-mediawiki01 was an hhvm core file; fixed now [23:39:02] Logged the message, Master [23:44:13] bd808: do we need to worry about HHVM coring? I thought that had stopped. [23:46:07] chrismcmahon: New bugs [23:46:25] I just got a stack trace and pasted it to ori [23:46:53] It's related to something he and Tim have been working on (tidy extension)