[00:03:21] 10Gitblit-Deprecate, 6Release-Engineering-Team, 10Diffusion, 7WorkType-NewFunctionality: Use Diffusion as canonical location for browsing code repos (not gitblit) - https://phabricator.wikimedia.org/T752#2116966 (10mmodell) [00:03:23] 5Gerrit-Migration, 10Diffusion: Linkify gerrit change-id in diffusion - https://phabricator.wikimedia.org/T89939#2116964 (10mmodell) 5Open>3Resolved [00:22:31] Krenair: do you still need a cloudadmin to fix something in beta cluster? [00:22:40] no [00:22:56] okey doke. just saw that in backscroll [00:22:56] I got Andrew to do it [00:25:29] I was moving all the mobile domains to the text cache [00:25:38] but the wikidata.beta.wmflabs.org domain was missing from the list [00:25:49] not sure why those beta.wmflabs.org are labs-global [01:06:02] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #562: 04FAILURE in 1.6 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/562/ [01:17:29] 10Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review, 7WorkType-NewFunctionality: redirect gerrit repo paths to diffusion callsigns - https://phabricator.wikimedia.org/T110607#2117001 (10mmodell) [01:17:57] 10Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review, 7WorkType-NewFunctionality: redirect gerrit repo paths to diffusion callsigns - https://phabricator.wikimedia.org/T110607#1582193 (10mmodell) D139 should allow us to close this? [01:20:23] 6Release-Engineering-Team, 10Phabricator, 6Project-Admins, 7WorkType-Maintenance: Create a Policy Admins project and move all of the acl*various_policy_admins pojects under it as subprojects. - https://phabricator.wikimedia.org/T129515#2117006 (10mmodell) [01:21:18] 6Release-Engineering-Team, 6Discovery, 10Wikimedia-Portals, 3Discovery-Portal-Sprint, 10Scap3 (Scap3-Adoption-Phase2): Deploy portals outside of mediawiki-config - https://phabricator.wikimedia.org/T129436#2117007 (10mmodell) [01:23:44] 6Release-Engineering-Team, 6Discovery, 10Wikimedia-Portals, 3Discovery-Portal-Sprint, 10Scap3 (Scap3-Adoption-Phase2): Deploy portals outside of mediawiki-config - https://phabricator.wikimedia.org/T129436#2117009 (10mmodell) [01:23:46] 10scap, 6Discovery, 10Scap3 (Scap3-Adoption-Phase1): Deploy wikimedia/portals with scap3 - https://phabricator.wikimedia.org/T114694#2117010 (10mmodell) [03:13:07] Yippee, build fixed! [03:13:08] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #910: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/910/ [07:33:05] 10Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review, 7WorkType-NewFunctionality: redirect gerrit repo paths to diffusion callsigns - https://phabricator.wikimedia.org/T110607#2117235 (10Paladox) We'll probably yes but when repos are created in gerrit they are not automatically added to this redirection sc... [08:33:28] Yippee, build fixed! [08:33:28] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #909: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/909/ [09:42:31] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer browser tests - https://phabricator.wikimedia.org/T129483#2107040 (10hashar) The browser test shell script we use set MEDIAWIKI_API_URL=http://... whic... [09:43:02] (03CR) 10Hashar: "Please do ! :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/276469 (https://phabricator.wikimedia.org/T129472) (owner: 10Zfilipin) [09:43:03] Yippee, build fixed! [09:43:04] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #747: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/747/ [09:50:02] 10Continuous-Integration-Config, 7I18n: Configure banana checker for i18n files to run on all MediaWiki extensions and skins (tracking) - https://phabricator.wikimedia.org/T94547#2117431 (10Nikerabbit) [09:59:31] 10Continuous-Integration-Config, 10Mathoid, 15User-mobrovac: Enable node4 for mathoid tests (mathoid/deploy tests fail) - https://phabricator.wikimedia.org/T124447#2117448 (10mobrovac) 5Open>3stalled @hashar not much I can do until I get access to the instances failing the jobs :/ [09:59:34] 5Continuous-Integration-Scaling, 6Services, 7Tracking, 7WorkType-NewFunctionality: [tracking] Migrate services CI jobs to node 4.3 - https://phabricator.wikimedia.org/T126991#2117450 (10mobrovac) [10:18:15] !log re-enabled systemd unit for logstash on deployment-logstash2 [10:18:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:24:46] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer browser tests - https://phabricator.wikimedia.org/T129483#2117493 (10zeljkofilipin) p:5Low>3Normal [10:40:31] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer browser tests - https://phabricator.wikimedia.org/T129483#2117500 (10zeljkofilipin) The problem is that in [[ https://phabricator.wikimedia.org/diffusio... [11:35:44] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:35] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 40242 bytes in 1.158 second response time [11:45:25] Yippee, build fixed! [11:45:26] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » en,contintLabsSlave && UbuntuTrusty build #90: 09FIXED in 9 min 3 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=en,label=contintLabsSlave%20&&%20UbuntuTrusty/90/ [11:58:09] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » mk,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 6 min 46 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=mk,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [12:00:30] 10MediaWiki-Codesniffer: Fix IfElseStructureSniff can't detect and fix multiple whitespace after else - https://phabricator.wikimedia.org/T129840#2117636 (10Lethexie) [12:05:26] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » vi,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=vi,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [12:11:42] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » fi,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=fi,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [12:17:59] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » hr,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 26 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=hr,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [12:22:18] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ka,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ka,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [12:30:52] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ja,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 39 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ja,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [12:30:56] (03PS1) 10Lethexie: Fix IfElseStructureSniff can't detect and fix multiple white spaces after else [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/277227 (https://phabricator.wikimedia.org/T129840) [12:34:46] 10scap, 10Citoid, 10VisualEditor, 10Scap3 (Scap3-Adoption-Phase1): Deploy Citoid with scap3 - https://phabricator.wikimedia.org/T116337#2117739 (10mobrovac) a:3mobrovac [12:35:02] 10MediaWiki-Codesniffer, 13Patch-For-Review: Fix IfElseStructureSniff can't detect and fix multiple white spaces after else - https://phabricator.wikimedia.org/T129840#2117740 (10Lethexie) [12:37:45] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » he,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 46 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=he,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [12:40:51] (03PS1) 10Reedy: Don't branch OAI anymore [tools/release] - 10https://gerrit.wikimedia.org/r/277231 (https://phabricator.wikimedia.org/T70867) [12:45:04] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » om,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 53 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=om,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [12:48:00] PROBLEM - Free space - all mounts on deployment-fluorine is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine.diskspace._srv.byte_percentfree (<20.00%) [12:50:59] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ps,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 59 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ps,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [12:56:23] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ru,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 1 hr 4 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ru,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [13:05:25] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » hy,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 1 hr 14 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=hy,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [13:11:47] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » es,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 1 hr 20 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=es,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [13:17:48] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » hu,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 1 hr 26 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=hu,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [13:29:49] oops [13:33:33] (03PS1) 10Schenkerx: Fix trailing spaces and convert spaces to tabs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/277235 (https://phabricator.wikimedia.org/T129474) [13:35:06] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » sco,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 1 hr 43 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=sco,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [13:36:41] (03CR) 10Schenkerx: "Hi," [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/276466 (https://phabricator.wikimedia.org/T129474) (owner: 10Schenkerx) [13:44:26] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ms,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 1 hr 53 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ms,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [13:51:25] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » lb,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 2 hr 0 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=lb,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [14:00:32] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » sr,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 2 hr 9 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=sr,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [14:05:27] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ar,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 2 hr 14 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ar,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [14:12:40] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ko,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 2 hr 21 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ko,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [14:20:28] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ast,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 2 hr 29 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ast,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [14:28:39] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » pt,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 2 hr 37 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=pt,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [14:37:41] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » fr,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 2 hr 46 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=fr,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [14:45:07] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ro,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 2 hr 53 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ro,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [14:51:13] PROBLEM - Host cache-rsync is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [14:52:11] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribuntu fails CI on 1.26 branch on php 5.3 - https://phabricator.wikimedia.org/T129854#2118067 (10JanZerebecki) [14:52:27] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » kn,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 3 hr 1 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=kn,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [14:52:42] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribuntu fails CI on 1.26 branch on php 5.3 - https://phabricator.wikimedia.org/T129854#2118067 (10JanZerebecki) Is this because of a missing PHP extension on our PHP 5.3 installation? Did this work before? [14:52:54] hrmmm [14:53:09] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribuntu fails CI on 1.26 branch on php 5.3 - https://phabricator.wikimedia.org/T129854#2118084 (10JanZerebecki) Is this because we changed the setting for the lua engine in CI? [14:53:43] 10Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review, 7WorkType-NewFunctionality: redirect gerrit repo paths to diffusion callsigns - https://phabricator.wikimedia.org/T110607#2118091 (10demon) That's not a problem, we just update the script :) [15:01:16] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » tr,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 3 hr 9 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=tr,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [15:07:59] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » br,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 3 hr 16 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=br,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [15:10:46] zeljkof: did we flip a switch on announcing browser test jobs here? [15:12:02] greg-g: no, as far as I know [15:15:01] so just a ton more failures than normal? [15:15:21] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » hi,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 3 hr 23 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=hi,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [15:18:48] 10Gitblit-Deprecate, 10Wikimedia-Apache-configuration: Write redirect rules from gitweb (to diffusion or other) - https://phabricator.wikimedia.org/T43961#2118416 (10JanZerebecki) [15:18:52] 10Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review, 7WorkType-NewFunctionality: redirect gerrit repo paths to diffusion callsigns - https://phabricator.wikimedia.org/T110607#2118413 (10JanZerebecki) 5Open>3Resolved [15:19:51] zeljkof: idea on why it seems to be more spammy today? [15:19:54] ideas* [15:22:52] greg-g: hm, we did fix visual editor screenshot job, that might be it [15:23:04] https://integration.wikimedia.org/ci/view/BrowserTests/view/VisualEditor/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/ [15:23:54] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » pl,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 3 hr 32 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=pl,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [15:31:21] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » sl,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 3 hr 39 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=sl,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [15:32:29] "fix" [15:32:30] :) [15:35:13] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is quite crowed (~200 k jobs) - https://phabricator.wikimedia.org/T128357#2118505 (10demon) If we don't actually have a ton of video transcodes I'd just go with 2 general purpose job runners and no dedicated ones. [15:40:15] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » id,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 3 hr 48 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=id,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [15:41:56] greg-g: is this getting closer to the versions dashboard you want? -- https://tools.wmflabs.org/bd808-test/versions.php [15:42:35] bd808: yup! [15:42:51] bd808: how was your weekend otherwise? :) [15:44:02] pretty good. The building we are remodeling is coming along pretty quickly and I binge watched The Wire. :) [15:45:27] 10Beta-Cluster-Infrastructure, 6Operations, 10Traffic, 13Patch-For-Review, 7Puppet: Fix puppet on deployment-cache* hosts in beta cluster - https://phabricator.wikimedia.org/T129270#2118591 (10greg) [15:45:35] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » bs,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 3 hr 54 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=bs,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [15:46:50] The groups there are actually using the dblist files by fetching them from noc. Kind of slow when developing remotely but seems plenty fast once it is deployed to tool labs [15:48:28] bd808: the building == the new law firm office? [15:48:49] bd808: nice, hopefully noc doesn't go away :) [15:50:10] yeah remodel is the new office building. framing is done and we did a walk through to make sure the electrical and plumbing stuff is in the right place. [15:50:48] If noc disappears for some reason I'd just need to clone mw-config myself. [15:51:02] but I really doubt noc will disappear [15:51:29] 10Continuous-Integration-Infrastructure: Give @mobrovac access to CI instances - https://phabricator.wikimedia.org/T129880#2118616 (10greg) [15:51:41] * greg-g nods [15:52:00] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » de,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 4 hr 0 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=de,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [15:53:59] bd808, greg-g: isn't there a ticket about noc going away? [15:54:16] Krenair: see, I thought I heard/saw something, I wasn't making that up out of the blue [15:54:27] I couldn't remember where I saw it, thought it was just rumblig on IRC [15:54:32] rumbling* [15:54:47] why? somebody crabby about symlink maintenance? [15:55:08] probably :) [15:55:24] Just because there's an open ticket just means someone hasn't closed it as invalid yet [15:55:36] * greg-g waits for Reedy to do it [15:55:48] T73723 [15:56:09] stashbot: hey! [15:56:22] grumble [15:56:25] a lot of the conf part of noc.wikimedia.org is like a git viewer [15:56:30] and there are like 3 of those available [15:56:42] Krenair: we like viewing files around here [15:56:55] greg-g: We should embed github in noc/conf [15:57:02] none of which view the clone state on the deploy server [15:57:11] Luke|away: best not to keep pushing on that task until you need to [15:57:31] bd808: point [15:57:48] Reedy: Don't go out of your way to get +q'ed. ;-) [15:57:55] greg-g: stashbot has a bad Phab session again. Apparently I need to put some code in to deal with that more gracefully [15:58:21] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » fa,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 4 hr 6 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=fa,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [15:58:21] eh? [15:58:27] I gave reedy op? [15:58:31] uh oh :) [15:59:51] T73723 [15:59:51] T73723: Move https://noc.wikimedia.org/conf/ to a wiki page on wikitech.wikimedia.org - https://phabricator.wikimedia.org/T73723 [16:04:29] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » cs,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 4 hr 13 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=cs,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [16:12:58] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » hsb,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 4 hr 21 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=hsb,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [16:16:43] 10Deployment-Systems, 10scap, 10Analytics, 10Analytics-Cluster, 10Scap3 (Scap3-Adoption-Phase1): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2096617 (10Milimetric) p:5Triage>3Normal [16:17:08] (03CR) 10Chad: [C: 032] make-wmf-branch: Branch UrlShortener extension [tools/release] - 10https://gerrit.wikimedia.org/r/276913 (https://phabricator.wikimedia.org/T108557) (owner: 10Legoktm) [16:18:38] (03Merged) 10jenkins-bot: make-wmf-branch: Branch UrlShortener extension [tools/release] - 10https://gerrit.wikimedia.org/r/276913 (https://phabricator.wikimedia.org/T108557) (owner: 10Legoktm) [16:19:17] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » it,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 4 hr 27 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=it,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [16:28:05] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » sv,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 4 hr 36 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=sv,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [16:31:45] 10Gitblit-Deprecate, 10Diffusion: Diffusion redirect from name to callsign doesn't always work - https://phabricator.wikimedia.org/T129447#2105968 (10demon) Fixed as part of {D139}, just needs deploying. [16:34:48] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ilo,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 4 hr 43 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ilo,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [16:35:59] 5Gerrit-Migration, 10Differential, 7Documentation, 7Upstream: Document best practices to amend a change written by another contributor - https://phabricator.wikimedia.org/T121751#2118860 (10mmodell) [16:37:27] 5Gerrit-Migration, 10Differential, 7Documentation, 7Upstream: Document best practices to amend a change written by another contributor - https://phabricator.wikimedia.org/T121751#1887024 (10mmodell) [16:38:14] It looks like https://integration.wikimedia.org/zuul/ has crashed because of an unkown repo. [16:38:25] jzerebecki Krinkle legoktm ^^^ [16:40:18] (03PS1) 10Paladox: [operations/software/logstash-logback-encoder] Add noob test [integration/config] - 10https://gerrit.wikimedia.org/r/277268 [16:40:57] 5Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#2118880 (10demon) `{{git file}}` is already using Diffusion, all the callsigns were updated in {D139}. What's left? [16:43:08] 5Gerrit-Migration, 10Differential, 7Documentation, 7Upstream: Document best practices to amend a change written by another contributor - https://phabricator.wikimedia.org/T121751#2118889 (10mmodell) There is a whole lot of discussion in upstream diff https://secure.phabricator.com/D15468 and task https://s... [16:44:50] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » et,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 4 hr 53 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=et,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [16:51:38] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » krc,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 5 hr 0 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=krc,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [16:53:38] (03PS1) 10Krinkle: Archive ArticleFeedbackv5 extension [integration/config] - 10https://gerrit.wikimedia.org/r/277275 [16:54:16] Krinkle zuul has crashed. [16:54:23] Fixed in https://gerrit.wikimedia.org/r/277268 [16:54:33] (03CR) 10Paladox: [C: 031] Archive ArticleFeedbackv5 extension [integration/config] - 10https://gerrit.wikimedia.org/r/277275 (owner: 10Krinkle) [16:58:02] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » el,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 5 hr 6 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=el,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [17:00:55] greg-g: Which task do you mean? [17:01:52] 10Continuous-Integration-Infrastructure, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2118983 (10Krinkle) [17:02:33] 10Continuous-Integration-Infrastructure, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2079136 (10Krinkle) 5Resolved>3Open This was not resolved. It was hacked around. Happening again: ``` 2016-03-14 17:00:28,100 ERROR zu... [17:02:59] !log Doing full Zuul restart due to deadlock (T128569) [17:03:00] T128569: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569 [17:03:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:03:04] 10Continuous-Integration-Infrastructure, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2079136 (10Paladox) Another crash happended because of an unkown project which I have added here https://gerrit.wikimedia.org/r/277268 [17:04:44] (03CR) 10Krinkle: [C: 04-1] "If it is at all feasible to fix this without this change then that is preferred. It is not acceptable to have to add no-ops for every repo" [integration/config] - 10https://gerrit.wikimedia.org/r/277268 (owner: 10Paladox) [17:06:45] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » zh-hant,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 5 hr 15 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=zh-hant,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [17:06:50] (03CR) 10Paladox: "@Krinkle yes but it is going to keep crashing. The support for non added projects in zuul needs to be supported upstream which I doint thi" [integration/config] - 10https://gerrit.wikimedia.org/r/277268 (owner: 10Paladox) [17:07:10] Luke081515: uninstalling conpherence [17:08:01] ah, ok [17:08:04] 10Continuous-Integration-Infrastructure, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2119012 (10Paladox) A bug needs to be filed upstream with high priority since this will affect everyone. [17:08:10] (03CR) 10Krinkle: "It is not a feature to not-crash when Gerrit receives a commit to a repo not known to Zuul. This used to work fine. This is a bug and regr" [integration/config] - 10https://gerrit.wikimedia.org/r/277268 (owner: 10Paladox) [17:09:13] 10Continuous-Integration-Infrastructure, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2119019 (10Krinkle) @hashar Zuul didn't used to crash when Gerrit receives a commit to a repo not known to Zuul. This used to work fine. Can... [17:09:29] (03CR) 10Paladox: "@Krinkle probally yes. Maybe @Hashar can test using a test installation of zuul on the master branch then test with some projects not adde" [integration/config] - 10https://gerrit.wikimedia.org/r/277268 (owner: 10Paladox) [17:14:11] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » yue,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 5 hr 22 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=yue,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [17:16:06] Krinkle i think i found the patch that fixes this upstream here https://github.com/openstack-infra/zuul/commit/0deaaadac7143692961b9d28abee8cea570ff3ce [17:16:23] 10Continuous-Integration-Infrastructure, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2119070 (10Paladox) I think its fixed upstream here https://github.com/openstack-infra/zuul/commit/0deaaadac7143692961b9d28abee8cea570ff3ce [17:20:08] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » nl,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 5 hr 28 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=nl,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [17:20:17] greg-g: Seems like the known users likes reopening :D [17:20:41] Luke081515: exactly [17:20:54] annonying... [17:23:57] 10Continuous-Integration-Infrastructure, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2119128 (10Paladox) Looks like we either have to wait for them to release there next version which may be a while or we can update straight... [17:27:19] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » nb,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 5 hr 35 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=nb,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [17:28:18] (03CR) 10Paladox: "Looks like it is fixed upstream https://github.com/openstack-infra/zuul/commit/0deaaadac7143692961b9d28abee8cea570ff3ce" [integration/config] - 10https://gerrit.wikimedia.org/r/277268 (owner: 10Paladox) [17:34:53] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » uk,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 5 hr 43 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=uk,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [17:37:55] 3Scap3, 10scap, 7Documentation, 15User-mobrovac: Document Scap3 config-deploy - https://phabricator.wikimedia.org/T116634#2119212 (10dduvall) >>! In T116634#2096765, @mmodell wrote: > I guess what I'm saying is, extra layers of abstraction don't help here, it just creates more work for each service to tran... [17:41:29] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » gl,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 5 hr 50 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=gl,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [17:49:45] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » zh-hans,contintLabsSlave && UbuntuTrusty build #91: 09SUCCESS in 5 hr 58 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=zh-hans,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [17:53:18] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » af,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 6 hr 1 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=af,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [17:59:35] 10Deployment-Systems, 3Scap3: Scap3 Service Restart Permissions - https://phabricator.wikimedia.org/T129897#2119320 (10thcipriani) [17:59:46] 10MediaWiki-Codesniffer, 7Easy: Add sniff to detect usage of deprecated $wgUser - https://phabricator.wikimedia.org/T50963#535102 (10Aashaka) I am interested in working on this. Should the PHP CodeSniffer sniff for $wgUser and warn to use RequestContext::getMain()->getUser() instead? Also, should it be automat... [18:01:15] Project browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox » ce,contintLabsSlave && UbuntuTrusty build #91: 04FAILURE in 6 hr 9 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox/LANGUAGE_SCREENSHOT_CODE=ce,label=contintLabsSlave%20&&%20UbuntuTrusty/91/ [18:03:16] (03PS1) 10Paladox: [AssertEdit] Archive extension [integration/config] - 10https://gerrit.wikimedia.org/r/277291 [18:12:54] (03PS1) 10Paladox: [Auth_remoteuser] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/277296 [18:18:52] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribuntu fails CI on 1.26 branch on php 5.3 - https://phabricator.wikimedia.org/T129854#2118067 (10Anomie) >>! In T129854#2118079, @JanZerebecki wrote: > Is this because of a missing PHP extension on our PHP 5.3 installat... [18:21:34] (03PS1) 10Paladox: [AuthorProtect] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/277300 [18:24:48] (03PS1) 10Paladox: [AutoCreateCategoryPages] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/277304 [18:26:39] 10Deployment-Systems, 3Scap3: Scap3 Service Restart Permissions - https://phabricator.wikimedia.org/T129897#2119493 (10thcipriani) This was called out as a security concern in the [[https://www.mediawiki.org/wiki/Deployment_tooling/Cabal/2016-03-14|deploy cabal meeting]]. Certainly, this has security implicat... [18:27:39] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:05] 6Release-Engineering-Team, 15User-greg: Outline roles and responsibilities for SWAT members - https://phabricator.wikimedia.org/T129905#2119500 (10greg) [18:28:16] (03CR) 10Jdlrobson: [C: 031] Set MEDIAWIKI_ENVIRONMENT environment variable for MultimediaViewer browsertests job [integration/config] - 10https://gerrit.wikimedia.org/r/276469 (https://phabricator.wikimedia.org/T129472) (owner: 10Zfilipin) [18:28:17] 6Release-Engineering-Team, 15User-greg: Outline roles and responsibilities for SWAT members - https://phabricator.wikimedia.org/T129905#2119516 (10greg) [18:30:55] 10Deployment-Systems, 3Scap3: Create `deploy-init` command for scap3 - https://phabricator.wikimedia.org/T129906#2119521 (10thcipriani) [18:32:32] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 29539 bytes in 0.863 second response time [18:33:51] (03PS1) 10Paladox: [AutomaticBoardWelcome] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/277309 [18:49:29] greg-g: is there a rate limiting on access to gerrit from one IP ? [18:50:57] * greg-g shrugs [18:50:58] ostriches: ^ [18:51:22] Nope [18:51:29] we are having a pre-hackathon, and suddenly we alll get timeout on access to gerrit [18:51:42] No IP limiting on our side there. [18:51:43] we get: [18:51:45] ssh: connect to host gerrit.wikimedia.org port 29418: Network is unreachable [18:51:45] fatal: Could not read from remote repository. [18:51:45] Please make sure you have the correct access rights [18:51:45] and the repository exists. [18:52:07] SSH on 29418 can be wonky on some networks that do port blocking. Have you tried over HTTPS? [18:52:22] checking [18:53:28] ostriches: no joy [18:53:38] Hrm. [19:03:31] (03PS1) 10Paladox: [AutoProxyBlock] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/277316 [19:06:08] ssh to gerrit works for me [19:06:46] (03PS1) 10Paladox: [Awesomeness] Add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/277320 [19:08:12] greg-g: https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=365465&oldid=365313 [19:09:47] legoktm: kk&ty [19:18:58] (03PS1) 10Paladox: [AWSSDK] Add npm test and add composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/277324 [19:42:38] 5Continuous-Integration-Scaling, 7Upstream: Nodepool instances have duplicate entry for jessie-backports/main - https://phabricator.wikimedia.org/T120963#2119793 (10hashar) Related bugs were T126230 and T126246 The fix is https://gerrit.wikimedia.org/r/#/c/270433/ 1d667fb6ccb24c83abc7dfcdecd2a52eae2c9ae6 . Th... [19:43:17] 5Continuous-Integration-Scaling, 7Upstream: Nodepool instances have duplicate entry for jessie-backports/main - https://phabricator.wikimedia.org/T120963#1865746 (10hashar) p:5Triage>3Normal [19:47:18] 10Gitblit-Deprecate, 10Diffusion: Update mediawiki.org templates to link to Diffusion, not gitblit - https://phabricator.wikimedia.org/T108864#2119820 (10mmodell) [19:47:20] 10Gitblit-Deprecate, 6Release-Engineering-Team, 10Diffusion, 7WorkType-NewFunctionality: Use Diffusion as canonical location for browsing code repos (not gitblit) - https://phabricator.wikimedia.org/T752#2119821 (10mmodell) [19:47:22] 5Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#2119818 (10mmodell) 5Open>3Resolved a:3mmodell [19:47:46] 5Gerrit-Migration, 10Gitblit-Deprecate: Update {{git file}} to link to diffusion - https://phabricator.wikimedia.org/T101358#1336469 (10mmodell) a:5mmodell>3None not sure who should claim this one [19:50:22] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:02:32] hashar: It looks like the problem where zuul crashes because of an unkown project has been fixed upstream https://github.com/openstack-infra/zuul/commit/0deaaadac7143692961b9d28abee8cea570ff3ce [20:02:37] https://phabricator.wikimedia.org/T128569 [20:04:11] paladox: good find! [20:05:44] (03PS1) 10Hoo man: Don't use --with-phpunitdir for Wikibase tests [integration/config] - 10https://gerrit.wikimedia.org/r/277330 [20:05:56] legoktm: hashar ^ could you take a look, please [20:07:10] 10Continuous-Integration-Infrastructure, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2119919 (10hashar) >>! In T128569#2119019, @Krinkle wrote: > @hashar Zuul didn't used to crash when Gerrit receives a commit to a repo not k... [20:07:19] 10Continuous-Integration-Infrastructure, 7Upstream, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2119920 (10hashar) [20:08:04] paladox: will get to review the patch then bump our Zuul version [20:09:07] hoo: I have added Jan as reviewer. i am too afraid to touch the wikibase jobs at 9pm my time :( [20:09:14] argh [20:09:18] hoo: Can you confirm it takes the "PHPUnit already present" path in the build? [20:09:19] they're broken [20:09:22] no [20:09:28] I have no access to the job runners [20:09:31] I'm just guessing [20:09:36] but it really should be [20:09:38] hoo: It's in the console output [20:09:47] The commit you reference in core shows it. [20:10:27] Oh, yes [20:10:30] right [20:11:39] hoo: looks like the wikibase do run composer :D one shows Installing phpunit/phpunit (4.8.23) [20:12:00] yeah, just saw [20:12:01] hoo: guess we can deploy them and see what happens? [20:12:05] sorry for not looking that up further [20:12:08] please [20:12:15] it can't be more broken than it is now :P [20:13:23] !log Updating Jenkins jobs mwext-Wikibase-* so they no more rely on --with-phpunit ( ping @hoo https://gerrit.wikimedia.org/r/#/c/277330/ ) [20:13:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:15:04] hoo: refreshed please verify :-} [20:15:05] hashar: Thanks, will test [20:15:42] (03CR) 10Hashar: "Jobs updated" [integration/config] - 10https://gerrit.wikimedia.org/r/277330 (owner: 10Hoo man) [20:16:04] * hashar watches https://integration.wikimedia.org/zuul/ [20:18:54] 20:17:15 PHPUnit 4.8.23 by Sebastian Bergmann and contributors. [20:18:58] looks good [20:19:50] hoo: I have no idea how we ended up being able to drop that --with-phpunit flag but the thing I know is that Jan worked hard on that front! [20:28:11] 10scap, 6Discovery, 10Scap3 (Scap3-Adoption-Phase1): Deploy wikimedia/portals with scap3 - https://phabricator.wikimedia.org/T114694#2120008 (10Deskana) [20:33:07] 10Continuous-Integration-Infrastructure, 7Upstream, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2120032 (10Paladox) Thanks. [20:34:25] (03CR) 10Hashar: [C: 032] "That fixed https://gerrit.wikimedia.org/r/#/c/276751/ :)" [integration/config] - 10https://gerrit.wikimedia.org/r/277330 (owner: 10Hoo man) [20:37:20] (03Merged) 10jenkins-bot: Don't use --with-phpunitdir for Wikibase tests [integration/config] - 10https://gerrit.wikimedia.org/r/277330 (owner: 10Hoo man) [20:42:36] 10Continuous-Integration-Infrastructure, 7Upstream, 7WorkType-Maintenance, 7Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2120057 (10Paladox) We may want to include all changes in the master branch because I think it includes a fix for something per... [20:48:36] hashar: we should include all the changes in the master branch when upgrading Zuul. Reasons because I think I've seen some patches that have fixed a performance issue, which I may think fixes the issue whereby sometimes it takes Zuul more then 5 seconds for it to detect a repo needs to be tested. But I'm not sure [20:51:18] paladox: I do a full review of all commits before bumping [20:51:38] some patches might have follow up fix [20:51:47] and other patches might need more testing for our setup [20:51:53] so I am playing it carefully [20:52:39] Oh ok thanks for explaining. [20:54:43] hashar: would this fix https://github.com/openstack-infra/zuul/commit/811e2e933440a66324be7bc37db36bf94395ab30 some issue we have when we have to wait along time for a change to be picked up [20:55:44] paladox: I dont think we run the connection change Joshua wrote [20:56:18] confirmed, we dont [20:56:23] hoo: I have merged the change [20:56:29] Oh ok, thanks for replying. [20:56:56] I wonder if that problem has been fixed then since I notice sometimes it takes longer then 5seconds [20:57:30] o/ greg-g [20:57:39] :) [20:57:49] so. yeah. Thinking of these large files that we want to keep versioned along-side code. [20:58:03] hashar: Thanks again! [20:58:03] Was looking into git LFS, but it sounds like that is a proprietary mess ATM. [20:58:37] yeah, git-fat is what the java services use right now with trebuchet (jarssssss), which we put support for in scap3 so they don't have to chagne much when migrating [20:58:55] git-annex appears (?) to be the current favorite https://phabricator.wikimedia.org/T119443 [20:59:00] Cool. Sounds like what we want. [20:59:04] though I haven't been in those discussions lately [20:59:06] Oh. Hmm. [21:00:14] Does git-fat do versioning? [21:01:49] greg-g, this is a place where I'd like to get a recommendation from the people who know best (probably you and your team) and invest into the technology. [21:02:18] thcipriani: ^ [21:02:40] tl;dr: what should new services use for managing large files if they want versioning for them? [21:04:08] FWIW, it looks like there are a few implementations of git LFS. https://github.com/github/git-lfs/wiki/Implementations [21:04:09] greg-g halfak: scap3 only supports git-fat atm (for parity w/ trebuchet), but we're planning support for git-annex as well [21:04:27] yeaht that ^ [21:04:37] from what we've evaluated so far, i think we're mostly in agreement that git-annex is the way to go [21:04:38] marxarelli, OK. Better to investin in git-fat or git-annex? [21:04:43] for The Future [21:04:44] kk [21:07:41] marxarelli & thcipriani, is there an annex server that I can start testing against? [21:09:12] halfak: afaik git annex isn't installed anywhere in the cluster. We do have it in our vagrant testing box: https://phabricator.wikimedia.org/diffusion/GSV/ [21:10:21] thcipriani, so, maybe I should just keep the blobs in git for now and deal with the massive repo size? [21:11:08] halfak: I'd say use git-fat for now. My understanding is that it's mostly rsync under the hood, which should be easy to translate to git-annex in future if need be. [21:11:30] halfak: i'm pretty sure you can use an rsync remote w/ git-annex without anything installed server side. is that right, thcipriani? [21:11:54] I'd like to keep mirroring on github. It seems like this might make that difficult. [21:12:40] marxarelli: correct [21:12:42] you can use rsync remote with git-annex without anything on the server (https://github.com/ocharles/papers for example) but you need git-annex on the client [21:12:47] git-annex can have rsync only remotes [21:13:06] * greg-g is a git-annex user for his personal use instead of Dropbox [21:13:57] Maybe we can just not have our config repo on github [21:14:01] That doesn't sound too crazy [21:14:10] Oh wait... we have other repos with blobs. [21:14:13] Nevermind. [21:16:11] halfak: you should be able to mirror without a problem if you're using an rsync remote. the blobs will just have to live somewhere centrally accessible, outside of the git repo itself [21:16:49] marxarelli, rsync over http? [21:28:31] halfak: what's the nature of the blobs that you want to store? are they read-only build artifacts or big files that contributors will need to access _and_ modify/push? [21:28:57] marxarelli, I want someone to be able to submit a pull request that includes a new binary. [21:30:09] They are model files -- trained prediction models that we use in revscoring to classify edits ad vandalism or to rank articles by their quality level. [21:30:32] But we'll also have python wheels for managing dependencies and data files for natural language processing. [21:30:33] halfak: i see. that's going to be a trickier use case than what most of our repos using git-fat require [21:31:01] afaik, they are mostly dealing with artifacts that are stored on archiva.wikimedia.org and then accessible read-only for git-fat to deploy [21:31:58] so, git-fat will ensure that the right versions of blobs are pulled down when deploying? [21:32:00] halfak: i'm not sure you can set up something that's as smooth as a pull request for the binary files, but you should shoot for a remote that you can easily grant access to for contributors [21:32:05] (because otherwise, why not just use a file-server) [21:32:13] maybe rsync with a remote on a labs instance or something [21:32:29] marxarelli, this is why I was looking at git-lfs which seamlessly integrates on github. [21:32:34] halfak: re git-fat deploy, yes [21:33:05] hmm. So we'd then manually copy from the labs instance to somewhere else that we'd allow our prod servers to reach? [21:34:29] halfak: unless ops/sec is ok with you pulling down binary files from labs for deploys, but i kind of doubt that [21:37:14] Seems clunky. Hmm. I think I'm just going to leave these things in the repo for now. [21:37:25] They aren't unmanageable yet. [21:37:31] But they will some day be so. [21:39:21] halfak: check out the other kinds of remotes that are supported by git-annex as well. maybe there's something there that would be better suited to your project. http://git-annex.branchable.com/special_remotes/ [21:39:56] marxarelli, already way over my quota for spending time on this :( [21:40:00] * halfak wears too many hats. [21:40:02] s3 maybe? i think you can set up an AWS policy that would allow for new objects to be written anonymously, but without allowing overwrites [21:40:12] But I promise that if I find time again, I'll be looking at git-annex [22:05:16] Anyone looking at Zuul? [22:11:16] new jobs aren't being triggered, huh? [22:12:17] lets see [22:15:22] the logs say it was Reedy's fault [22:15:28] It usually is [22:15:40] should be fixed now [22:16:56] heh [22:16:58] thanks legoktm [22:17:03] * greg-g kicks Reedy [22:17:18] hoo: better now thanks to lego [22:17:25] heh [22:17:40] !log new jobs weren't processing in Zuul, lego fixed it and blamed Reedy [22:17:57] logbot! [22:18:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:18:08] Reedy: do you want to file a bug for the circular deps? [22:18:23] Yeah, I think it's my responsibility :) [22:20:21] 10Continuous-Integration-Infrastructure, 7Zuul: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2120489 (10Reedy) [22:21:25] well that sucks [22:21:58] I presume an A-B-C-A will do the same [22:38:20] 6Release-Engineering-Team, 7Epic: [EPIC] (WIP) End-to-end tests and deploys - https://phabricator.wikimedia.org/T129940#2120532 (10greg) [22:46:38] 6Release-Engineering-Team, 15User-greg: Find a home for the "end-to-end and deploys" idea - https://phabricator.wikimedia.org/T129718#2120589 (10greg) 5Open>3Resolved [23:15:40] 6Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 6Operations: Intermittent read-only errors on s3 wikis on March 14th - https://phabricator.wikimedia.org/T129947#2120701 (10greg)