[00:21:17] 10Beta-Cluster: Extension Echo don't works on beta dewiki anymore - https://phabricator.wikimedia.org/T105977#1455859 (10Luke081515) 3NEW [00:44:25] 10Continuous-Integration-Infrastructure: Write and implement tests for Wikimedia's Apache configuration (redirects.conf, etc.) - https://phabricator.wikimedia.org/T45266#1455898 (10MZMcBride) [00:54:34] imma gonna mess with scap in beta cluster for a bit. [00:57:56] I think you broke it [00:58:26] I didn't start yet :) [00:58:33] is it still broke from earlier? [00:58:59] yeah [00:59:14] job is green -- https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ [00:59:39] is sync-file supposed to not work? [00:59:52] in beta? hmmm [01:00:25] it should work but you have to hold your mount just right. It's not setup for any user other than jenkins to do things as I recall [01:00:39] or some user that jenkins sudo's to [01:01:08] I tried it as jenkins-deploy and it still errored [01:05:09] Project beta-scap-eqiad build #61595: FAILURE in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/61595/ [01:11:57] 6Release-Engineering, 6Analytics-Backlog, 6Performance-Team, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1455975 (10mmodell) Krinkle: I don't think anyone has been removing branches prematurely. Also, I am pro... [01:17:56] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL 40.00% of data above the critical threshold [0.0] [01:22:33] 10Beta-Cluster: Extension Echo don't works on beta dewiki anymore - https://phabricator.wikimedia.org/T105977#1455996 (10Krenair) a:3bd808 Scap on beta was broken until Bryan fixed it a few minutes ago. It's fixed and so is this bug... I assume it was running broken code which is now solved. [01:22:40] 10Beta-Cluster: Extension Echo don't works on beta dewiki anymore - https://phabricator.wikimedia.org/T105977#1455998 (10Krenair) 5Open>3Resolved [01:23:40] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 20.00% of data above the critical threshold [0.0] [01:26:46] RECOVERY - Puppet staleness on deployment-restbase01 is OK Less than 1.00% above the threshold [3600.0] [01:26:56] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 60.00% of data above the critical threshold [0.0] [01:36:55] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [01:45:48] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1456014 (10Tgr) >>! In T102566#1451278, @Nemo_bis wrote: > The necessary steps also must be documented. At least one... [01:46:03] Yippee, build fixed! [01:46:03] Project beta-scap-eqiad build #61597: FIXED in 21 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/61597/ [01:48:50] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1456021 (10Tgr) >>! In T102566#1456014, @Tgr wrote: > I don't see anything needing change on any of those. Probably... [01:58:39] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [02:32:16] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1456072 (10hoo) p:5Normal>3High We plan to use the browser tests to test after the Tuesday deployments on testwikidata, but that wont wor... [02:34:13] !log cherry-picked https://gerrit.wikimedia.org/r/#/c/224313 for scap testing [02:34:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:35:43] Yippee, build fixed! [02:35:43] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #634: FIXED in 2 min 41 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/634/ [02:40:48] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1456093 (10hoo) [02:43:11] (03PS1) 10JanZerebecki: Change time of Wikidata browser test to run at 01:00 UTC [integration/config] - 10https://gerrit.wikimedia.org/r/224998 (https://phabricator.wikimedia.org/T105985) [02:47:42] 10Beta-Cluster, 5Patch-For-Review: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#1456102 (10jayvdb) Sorry for the silly question, but ... if that change [[https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/2919/console|has been deployed]] to t... [03:11:03] 10Beta-Cluster, 5Patch-For-Review: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#1456117 (10Krenair) >>! In T69931#1456102, @jayvdb wrote: > Sorry for the silly question, but ... if that change [[https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eq... [03:18:57] 10Deployment-Systems: Investigate what changes are needed to deploy MW+Extensions by percentage of users (instead of by domain/wiki) - https://phabricator.wikimedia.org/T104398#1456131 (10mmodell) The multiversion router could certainly be configured to key off of something other than the host: header. It would... [03:23:38] Yippee, build fixed! [03:23:38] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #752: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/752/ [03:32:32] 10Beta-Cluster, 5Patch-For-Review: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#1456167 (10Reedy) An interwiki.cdb for beta can't be generated (and expected to work correctly) until T75906 is fixed [03:37:58] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 10pywikibot-core, 5Patch-For-Review: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI - https://phabricator.wikimedia.org/T100903#1456179 (10jayvdb) [03:38:00] 10Beta-Cluster, 5Patch-For-Review: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#1456177 (10jayvdb) 5Resolved>3Open Re-opening as this has incomplete sub-tasks, and has tasks blocked on this being fixed. [04:09:45] Project beta-scap-eqiad build #61610: FAILURE in 25 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/61610/ [04:13:19] Yippee, build fixed! [04:13:19] Project beta-scap-eqiad build #61611: FIXED in 2 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/61611/ [04:34:12] Project beta-code-update-eqiad build #64156: FAILURE in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/64156/ [05:29:10] 10Deployment-Systems, 6Release-Engineering: No auto-submodule updates to 1.26wmf14 - https://phabricator.wikimedia.org/T105994#1456317 (10Krenair) 3NEW a:320after4 [05:35:39] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #481: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/481/ [05:36:08] 10Beta-Cluster, 10MediaWiki-extensions-WikimediaMaintenance: Check/update dumpInterwiki for beta usage - https://phabricator.wikimedia.org/T75906#1456327 (10Krenair) [05:36:10] 10Beta-Cluster, 5Patch-For-Review: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#1456326 (10Krenair) [05:36:31] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 10pywikibot-core, 5Patch-For-Review: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI - https://phabricator.wikimedia.org/T100903#1456329 (10Krenair) [05:36:35] 10Beta-Cluster, 5Patch-For-Review: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#1456328 (10Krenair) 5Open>3Resolved [06:01:12] 10Beta-Cluster, 6Labs, 10Labs-Infrastructure, 7Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#1456340 (10zhuyifei1999) [06:10:31] 10Beta-Cluster, 5Patch-For-Review: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#1456367 (10jayvdb) Fwiw, the tasks depending on this task do not depend on the issues raised in T75906#1011937 , as Pywikibot only chokes on the fact that the beta interwikimap link the l... [06:12:31] 10Beta-Cluster, 5Patch-For-Review: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#1456370 (10Krenair) We did run updateinterwikicache, and now there is an interwiki map (rather than nothing, and coming from interwiki-labs.cdb), but it has links to production sites, whi... [06:27:18] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #681: FAILURE in 8 min 17 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/681/ [06:33:45] 10Beta-Cluster, 10MediaWiki-extensions-WikimediaMaintenance: Check/update dumpInterwiki for beta usage - https://phabricator.wikimedia.org/T75906#1456391 (10jayvdb) [06:33:48] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 10pywikibot-core, 5Patch-For-Review: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI - https://phabricator.wikimedia.org/T100903#1456390 (10jayvdb) [06:33:54] 10Beta-Cluster, 10MediaWiki-extensions-WikimediaMaintenance: Check/update dumpInterwiki for beta usage - https://phabricator.wikimedia.org/T75906#784930 (10jayvdb) [06:46:29] RECOVERY - Free space - all mounts on deployment-videoscaler01 is OK All targets OK [06:53:29] Project UploadWizard-api-commons.wikimedia.org build #1780: FAILURE in 27 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/1780/ [06:59:02] PROBLEM - Puppet failure on integration-zuul-server is CRITICAL 100.00% of data above the critical threshold [0.0] [08:20:00] (03CR) 10Paladox: [C: 031] Add jshint test for metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/224834 (owner: 10Paladox) [09:05:57] 10Deployment-Systems, 6Release-Engineering, 7Epic: Rethinking mediawiki deployment process - https://phabricator.wikimedia.org/T89945#1456507 (10mmodell) [09:08:25] 10Deployment-Systems, 6Release-Engineering, 7Epic: Rethinking mediawiki deployment branches and release process - https://phabricator.wikimedia.org/T89945#1456524 (10mmodell) [09:10:17] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1456526 (10mmodell) See also {T102991} and ultimately {T89945} [09:12:16] 10Deployment-Systems: Automatically clean up unused wmfXX versions - https://phabricator.wikimedia.org/T73313#1456529 (10mmodell) [09:12:18] 10Deployment-Systems, 6Release-Engineering, 7Epic: Rethinking mediawiki deployment branches and release process - https://phabricator.wikimedia.org/T89945#1456530 (10mmodell) [09:15:31] 10Browser-Tests, 6Release-Engineering: Track state of browser tests before new wmfXX branch cut - https://phabricator.wikimedia.org/T437#1456531 (10mmodell) If we change to persistent release branches then we can simply look at the build status of individual commits as they get merged into the branch. [09:21:33] (03CR) 10Addshore: [C: 031] Change time of Wikidata browser test to run at 01:00 UTC [integration/config] - 10https://gerrit.wikimedia.org/r/224998 (https://phabricator.wikimedia.org/T105985) (owner: 10JanZerebecki) [09:23:39] 6Release-Engineering, 6Phabricator: Evaluate Releeph for SWAT queue management - https://phabricator.wikimedia.org/T101794#1456535 (10mmodell) So releeph is pretty close to what we want but I don't know if it's going to be ready for prime time, any time soon. I actually think we could just use differential or... [09:43:54] Yippee, build fixed! [09:43:55] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #548: FIXED in 6 min 53 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/548/ [09:48:39] (03CR) 10Hashar: [C: 031] "Feel free to rebase/deploy at any time :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/224998 (https://phabricator.wikimedia.org/T105985) (owner: 10JanZerebecki) [10:22:15] 10Beta-Cluster, 6operations: deployment-bastion fails puppet because of bacula - https://phabricator.wikimedia.org/T106003#1456624 (10hashar) 3NEW [10:26:44] !log deployment-bastion: apt-get upgrade [10:26:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:27:47] !log fixing puppet on deployment-bastion. Stalled since July 7th - https://phabricator.wikimedia.org/T106003 [10:27:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:27:56] (03PS6) 10Paladox: Add jshint test for metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/224834 [10:29:33] (03PS8) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [10:30:13] 10Beta-Cluster, 6operations, 5Patch-For-Review: deployment-bastion fails puppet because of bacula - https://phabricator.wikimedia.org/T106003#1456648 (10hashar) Did some cleanup based on `/var/log/apt/history.log` ``` Start-Date: 2015-07-07 22:35:01 Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--f... [10:30:51] 10Beta-Cluster, 6operations, 5Patch-For-Review: deployment-bastion fails puppet because of bacula - https://phabricator.wikimedia.org/T106003#1456650 (10hashar) [10:32:31] (03PS7) 10Paladox: Add jshint to vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [10:33:20] 10Beta-Cluster, 6operations, 5Patch-For-Review: deployment-bastion fails puppet because some classes were moved from nodes to role class - https://phabricator.wikimedia.org/T106003#1456660 (10hashar) [10:34:55] 10Beta-Cluster, 6operations, 5Patch-For-Review: deployment-bastion fails puppet because some classes were moved from nodes to role class - https://phabricator.wikimedia.org/T106003#1456624 (10hashar) [10:38:21] 10Beta-Cluster, 6operations, 5Patch-For-Review: deployment-bastion fails puppet because some classes were moved from nodes to role class - https://phabricator.wikimedia.org/T106003#1456673 (10hashar) a:3hashar [10:41:47] 10Beta-Cluster, 6operations, 5Patch-For-Review: deployment-bastion fails puppet because some classes were moved from nodes to role class - https://phabricator.wikimedia.org/T106003#1456674 (10hashar) p:5Triage>3Normal I have cherry picked both patches on beta cluster puppetmaster and puppet now passes on... [10:46:16] 6Release-Engineering, 6Team-Practices: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1456676 (10hashar) Yup @demon definitely took the lead and we have weekly triage since early June. We had a few iterations to refine the dashboard column on #wikimedia-log-errors and killed off m... [10:55:09] RECOVERY - Puppet failure on deployment-bastion is OK Less than 1.00% above the threshold [0.0] [10:55:26] (03PS3) 10Hashar: Remove some uses of Gerrit replications [integration/config] - 10https://gerrit.wikimedia.org/r/224814 [10:55:28] (03PS3) 10Hashar: Remove uses of Gerrit replicas in JJB defaults [integration/config] - 10https://gerrit.wikimedia.org/r/224818 [10:55:30] (03PS2) 10Hashar: Drop unused use-zuul and git-localonly-zuul [integration/config] - 10https://gerrit.wikimedia.org/r/224821 [11:00:27] (03CR) 10Hashar: [C: 032] "Jobs refreshed" [integration/config] - 10https://gerrit.wikimedia.org/r/224814 (owner: 10Hashar) [11:01:21] (03CR) 10Hashar: [C: 032] "Jobs deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/224818 (owner: 10Hashar) [11:02:04] (03CR) 10Hashar: [C: 032] "Diff job shows that this is a noop. Bye bye legacy stuff." [integration/config] - 10https://gerrit.wikimedia.org/r/224821 (owner: 10Hashar) [11:02:59] (03Merged) 10jenkins-bot: Remove some uses of Gerrit replications [integration/config] - 10https://gerrit.wikimedia.org/r/224814 (owner: 10Hashar) [11:03:40] (03Merged) 10jenkins-bot: Remove uses of Gerrit replicas in JJB defaults [integration/config] - 10https://gerrit.wikimedia.org/r/224818 (owner: 10Hashar) [11:04:12] (03Merged) 10jenkins-bot: Drop unused use-zuul and git-localonly-zuul [integration/config] - 10https://gerrit.wikimedia.org/r/224821 (owner: 10Hashar) [11:04:25] 10Deployment-Systems: Investigate what changes are needed to deploy MW+Extensions by percentage of users (instead of by domain/wiki) - https://phabricator.wikimedia.org/T104398#1456686 (10BBlack) >>! In T104398#1456131, @mmodell wrote: > On IRC, @BBlack suggested that we might consider lowering our cache TTL to... [11:06:13] (03PS4) 10Hashar: tests: precompile some regex [integration/config] - 10https://gerrit.wikimedia.org/r/224417 [11:06:31] (03CR) 10Hashar: [C: 032] tests: precompile some regex [integration/config] - 10https://gerrit.wikimedia.org/r/224417 (owner: 10Hashar) [11:07:04] (03CR) 10Hashar: tests: consistency of Zuul pipelines declarations [integration/config] - 10https://gerrit.wikimedia.org/r/224517 (owner: 10Hashar) [11:08:26] (03Merged) 10jenkins-bot: tests: precompile some regex [integration/config] - 10https://gerrit.wikimedia.org/r/224417 (owner: 10Hashar) [11:11:39] (03PS1) 10Paladox: Update tests in vector extension [integration/config] - 10https://gerrit.wikimedia.org/r/225029 [11:12:03] (03PS2) 10Paladox: Update tests in vector extension [integration/config] - 10https://gerrit.wikimedia.org/r/225029 [11:13:42] (03PS8) 10Paladox: Add jshint to vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [11:19:45] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1456697 (10BBlack) Re: caching, I think in the ideal variant of these ideas, we're not arbitrarily bu... [11:21:40] (03PS9) 10Paladox: Add jshint to vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [11:22:04] (03CR) 10Paladox: "CentralAuth does the same thing." [integration/config] - 10https://gerrit.wikimedia.org/r/224824 (owner: 10Paladox) [11:24:56] 10Deployment-Systems: Investigate what changes are needed to deploy MW+Extensions by percentage of users (instead of by domain/wiki) - https://phabricator.wikimedia.org/T104398#1456723 (10mmodell) @bblack: I think we should do essentially what you suggest, but instead of creating new branches each week we just h... [11:25:28] (03PS7) 10Paladox: Add jshint test for metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/224834 [11:26:01] (03PS8) 10Paladox: Update tests for Metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/224834 [11:26:34] (03PS9) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [11:27:42] (03PS10) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [11:28:24] (03PS6) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [11:29:22] (03PS7) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [11:54:53] (03PS1) 10Hashar: integration/jenkins migrated to tox [integration/config] - 10https://gerrit.wikimedia.org/r/225032 [11:55:05] (03CR) 10Hashar: [C: 032] integration/jenkins migrated to tox [integration/config] - 10https://gerrit.wikimedia.org/r/225032 (owner: 10Hashar) [11:56:06] (03PS2) 10Hashar: integration/jenkins migrated to tox [integration/config] - 10https://gerrit.wikimedia.org/r/225032 [11:56:13] (03CR) 10Hashar: [C: 032] integration/jenkins migrated to tox [integration/config] - 10https://gerrit.wikimedia.org/r/225032 (owner: 10Hashar) [11:58:21] (03Merged) 10jenkins-bot: integration/jenkins migrated to tox [integration/config] - 10https://gerrit.wikimedia.org/r/225032 (owner: 10Hashar) [12:02:30] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<50.00%) [12:07:16] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [0.0] [12:11:53] (03PS2) 10Hashar: add mediawiki/extensions/PhpTagsStorage [integration/config] - 10https://gerrit.wikimedia.org/r/224446 (owner: 10Pastakhov) [12:13:03] (03CR) 10Hashar: [C: 032] add mediawiki/extensions/PhpTagsStorage [integration/config] - 10https://gerrit.wikimedia.org/r/224446 (owner: 10Pastakhov) [12:15:31] (03CR) 10jenkins-bot: [V: 04-1] add mediawiki/extensions/PhpTagsStorage [integration/config] - 10https://gerrit.wikimedia.org/r/224446 (owner: 10Pastakhov) [12:15:39] (03CR) 10Hashar: [C: 032] add mediawiki/extensions/PhpTagsStorage [integration/config] - 10https://gerrit.wikimedia.org/r/224446 (owner: 10Pastakhov) [12:17:54] (03Merged) 10jenkins-bot: add mediawiki/extensions/PhpTagsStorage [integration/config] - 10https://gerrit.wikimedia.org/r/224446 (owner: 10Pastakhov) [12:25:35] (03PS1) 10Paladox: Update SyntaxHighlight_GeSHi tests [integration/config] - 10https://gerrit.wikimedia.org/r/225035 [12:26:11] (03PS2) 10Paladox: Update SyntaxHighlight_GeSHi tests [integration/config] - 10https://gerrit.wikimedia.org/r/225035 [12:26:51] (03PS3) 10Paladox: Update SyntaxHighlight_GeSHi tests [integration/config] - 10https://gerrit.wikimedia.org/r/225035 [12:29:33] (03PS8) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [12:37:35] (03PS3) 10Paladox: Update tests in vector extension [integration/config] - 10https://gerrit.wikimedia.org/r/225029 [12:53:30] Yippee, build fixed! [12:53:31] Project UploadWizard-api-commons.wikimedia.org build #1781: FIXED in 29 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/1781/ [13:09:12] (03PS1) 10Paladox: Update farmer tests [integration/config] - 10https://gerrit.wikimedia.org/r/225042 [13:09:59] (03PS2) 10Paladox: Update farmer tests [integration/config] - 10https://gerrit.wikimedia.org/r/225042 [13:13:51] (03PS3) 10Paladox: Update farmer tests [integration/config] - 10https://gerrit.wikimedia.org/r/225042 [13:14:25] (03CR) 10Sbisson: "From environment.rb:441" [selenium] - 10https://gerrit.wikimedia.org/r/224809 (owner: 10Sbisson) [13:17:47] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL 40.00% of data above the critical threshold [43200.0] [13:48:38] (03PS9) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [13:50:09] 10Beta-Cluster: Beta dewiki is now in english language - https://phabricator.wikimedia.org/T106019#1457014 (10Luke081515) 3NEW [13:58:00] https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/64212/console beta sync still failing [13:58:05] thcipriani iiiiii [13:58:18] marktraceur: looking [14:00:30] (03PS10) 10Paladox: Update theme tests [integration/config] - 10https://gerrit.wikimedia.org/r/224838 [14:06:04] (03PS4) 10Paladox: Update tests in vector extension [integration/config] - 10https://gerrit.wikimedia.org/r/225029 [14:06:57] Yippee, build fixed! [14:06:57] Project beta-code-update-eqiad build #64214: FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/64214/ [14:07:12] ^ marktraceur fixed [14:09:38] (03PS4) 10Paladox: Update SyntaxHighlight_GeSHi tests [integration/config] - 10https://gerrit.wikimedia.org/r/225035 [14:11:42] Yay! Thanks. [14:13:40] And now to debug. :) [14:16:51] how did I break it this time? [14:17:19] * bd808 is burning stocked up beta cluster karma pretty fast this week apparently [14:28:03] there was just some uncommited change to dumpInterwiki.php in the wikimediamaintenance extension [14:31:29] ah. that was Krenair last night debugging the iw cache [14:33:53] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #566: FAILURE in 7 min 52 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/566/ [14:38:13] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #193: FAILURE in 10 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/193/ [14:49:04] 10Beta-Cluster: Beta dewiki is now in english language - https://phabricator.wikimedia.org/T106019#1457222 (10Aklapper) Link highly welcome in bug reports... http://de.wikipedia.beta.wmflabs.org/ ? That's in German here when I'm not logged in. [14:49:38] Are duvall and antoine both at wikimania? [14:55:26] (03CR) 10Hashar: [C: 031] Publish phpunit coverage for utfnormal, at-ease & AhoCorasick [integration/config] - 10https://gerrit.wikimedia.org/r/220937 (owner: 10Legoktm) [15:01:04] hashar: I just emailed; are you working on nodepool today? [15:05:20] hashar: because nodepool is spamming me like crazy! [15:11:26] 10Beta-Cluster: Central login at betawiki doesn't work - https://phabricator.wikimedia.org/T106035#1457257 (10MGChecker) 3NEW [15:12:38] 10Beta-Cluster: Beta dewiki is now in english language - https://phabricator.wikimedia.org/T106019#1457274 (10MGChecker) Mee too, but if I am logged in, it is in English too, and I haven't changed my language setiings, so there has to be any bug. [15:13:12] 10Continuous-Integration-Infrastructure: Migrate mediawiki-core-code-coverage job to labs - https://phabricator.wikimedia.org/T93559#1457277 (10hashar) a:3hashar [15:13:36] (03PS1) 10Hashar: Migrate mediawiki-core-code-coverage job to labs [integration/config] - 10https://gerrit.wikimedia.org/r/225063 (https://phabricator.wikimedia.org/T93559) [15:14:12] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL 100.00% of data above the critical threshold [0.0] [15:14:19] 10Beta-Cluster: Central login at betawiki doesn't work - https://phabricator.wikimedia.org/T106035#1457284 (10MGChecker) [15:15:09] (03CR) 10Hashar: [C: 04-2] "WIP play testing with https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-2" [integration/config] - 10https://gerrit.wikimedia.org/r/225063 (https://phabricator.wikimedia.org/T93559) (owner: 10Hashar) [15:15:22] 10Beta-Cluster: Central login at betawiki doesn't work - https://phabricator.wikimedia.org/T106035#1457257 (10MGChecker) If this can help, I use Firefox 39 at Windows 8.1. [15:16:12] hashar: btw, if you emailed back I won’t notice because my inbox is so full of alerts :) [15:20:26] andrewbogott: oh I stopped using email entirely [15:20:41] I now rely exclusively on paper mail. That is so quiet nowadays [15:20:59] seems like the right choice [15:23:38] hashar: so… have a cron running every 10 seconds on nodepool? That’s about how often I’m getting "nodepool : user NOT in sudoers" [15:23:53] doh [15:24:00] yeah replying [15:24:04] the nodepool daemon is on [15:24:10] and indeed attempts to rebuild diskimages [15:25:23] Project beta-scap-eqiad build #61622: FAILURE in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/61622/ [15:30:57] andrewbogott: thanks for the notice. I am fixing it and writing down what I did so Dan/Zeljko now about it [15:31:17] hashar: thanks. My inbox has quieted down so something is better :) [15:31:29] yeah I stopped the service [15:31:34] there were no more anymore images defined [15:31:43] so nodepoold tried to rebuild one over and over :-( [15:33:01] I gotta find a way to build the image without needing any root access -:-\ [15:35:15] Yippee, build fixed! [15:35:15] Project beta-scap-eqiad build #61623: FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/61623/ [15:35:51] (03PS2) 10BryanDavis: [WIP] Sync /srv/mediawiki-staging to co-masters [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) [15:38:52] 10Beta-Cluster: Beta dewiki is now in english language - https://phabricator.wikimedia.org/T106019#1457381 (10Aklapper) Confirming that "default" strings are in English on http://de.wikipedia.beta.wmflabs.org/wiki/Wikipedia:Hauptseite after logging in. Loading http://de.wikipedia.beta.wmflabs.org/wiki/Wikipedia... [15:40:14] 10Beta-Cluster: Beta dewiki's strings are in English when logged in (does not happen in debug mode or other languages) - https://phabricator.wikimedia.org/T106019#1457386 (10Aklapper) [15:48:38] 10Browser-Tests, 10VisualEditor, 10Wikimania-Hackathon-2015: add true internationalized content support to auto-screenshots - https://phabricator.wikimedia.org/T105466#1457415 (10Amire80) p:5Triage>3High [15:58:37] (03PS3) 10BryanDavis: [WIP] Sync /srv/mediawiki-staging to co-masters [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) [16:01:22] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1457516 (10JanZerebecki) Git uses content addressing for files, so instead of using the last commit t... [16:02:32] 10Beta-Cluster, 7I18n: Beta dewiki's strings are in English when logged in (does not happen in debug mode or other languages) - https://phabricator.wikimedia.org/T106019#1457518 (10greg) [16:04:14] 10Beta-Cluster, 7I18n: Beta dewiki's strings are in English when logged in (does not happen in debug mode or other languages) - https://phabricator.wikimedia.org/T106019#1457525 (10MGChecker) [[ http://de.wikipedia.beta.wmflabs.org/wiki/Wikipedia:Hauptseite?uselang=de ]] gives the same strings as [[ http://de... [16:07:32] (03PS2) 10Hashar: Migrate mediawiki-core-code-coverage job to labs [integration/config] - 10https://gerrit.wikimedia.org/r/225063 (https://phabricator.wikimedia.org/T93559) [16:07:58] (03CR) 10Hashar: "More or less deployed. Will follow up tomorrow" [integration/config] - 10https://gerrit.wikimedia.org/r/225063 (https://phabricator.wikimedia.org/T93559) (owner: 10Hashar) [16:08:27] !log kept nodepool stopped on labnodepool1001.eqiad.wmnet because it spams the cron log [16:08:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:10:14] 10Beta-Cluster, 10Deployment-Systems, 7I18n: Beta dewiki's strings are in English when logged in (does not happen in debug mode or other languages) - https://phabricator.wikimedia.org/T106019#1457555 (10greg) [16:20:07] 10Deployment-Systems, 6Release-Engineering, 5Patch-For-Review: No auto-submodule updates to 1.26wmf14 - https://phabricator.wikimedia.org/T105994#1457657 (10mmodell) The reason for the inconsistency is that git ~v1.7 (the version we have on tin) doesn't set the `branch=wmf/$version` for the submodules in `.g... [16:21:02] (03CR) 10BryanDavis: [WIP] Sync /srv/mediawiki-staging to co-masters (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) (owner: 10BryanDavis) [16:25:01] Project beta-scap-eqiad build #61628: FAILURE in 20 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/61628/ [16:26:13] 10Deployment-Systems, 6Release-Engineering, 7Epic: Rethinking mediawiki deployment branches and release process - https://phabricator.wikimedia.org/T89945#1457679 (10Krinkle) What is the expected timeline on using long-lived/re-used branches in production? Please beware that moving to that system must be bl... [16:27:08] 10Beta-Cluster, 10MediaWiki-extensions-CentralAuth: Central login at betawiki doesn't work - https://phabricator.wikimedia.org/T106035#1457686 (10MGChecker) [16:27:43] Yippee, build fixed! [16:27:44] Project beta-scap-eqiad build #61629: FIXED in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/61629/ [16:38:58] 10Continuous-Integration-Infrastructure: Request Jenkins shell access for account "sniedzielski" - https://phabricator.wikimedia.org/T103192#1457734 (10Dzahn) @hashar For some reason he doesn't have a home directory yet. How did you grant him access? root@integration-slave-trusty-1015:/home# ls awight dduval... [16:39:09] (03PS1) 10Paladox: Add tests for WikibaseRepository [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T106002) [16:39:12] 10Deployment-Systems, 6Release-Engineering, 7Epic: Rethinking mediawiki deployment branches and release process - https://phabricator.wikimedia.org/T89945#1457735 (10mmodell) @krinkle: I don't know a specific timeline, but I'd like to hear about your ideas for solving {T99096} [16:39:22] 10Deployment-Systems, 6Release-Engineering, 7Epic: Rethinking mediawiki deployment branches and release process - https://phabricator.wikimedia.org/T89945#1457737 (10mmodell) [16:39:24] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1457738 (10mmodell) [16:39:31] (03PS2) 10Paladox: Add tests for WikibaseRepository [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T106002) [16:40:15] 10Deployment-Systems, 6Release-Engineering, 7Epic: Rethinking mediawiki deployment branches and release process - https://phabricator.wikimedia.org/T89945#1049275 (10mmodell) [16:40:17] 10Deployment-Systems, 6Release-Engineering: Use subrepos instead of git submodules for deployed MediaWiki extensions - https://phabricator.wikimedia.org/T98834#1457741 (10mmodell) [16:40:48] 10Deployment-Systems, 6Release-Engineering, 7Epic: Rethinking mediawiki deployment branches and release process - https://phabricator.wikimedia.org/T89945#1049275 (10mmodell) [16:40:51] 10Deployment-Systems, 6Release-Engineering: Use subrepos instead of git submodules for deployed MediaWiki extensions - https://phabricator.wikimedia.org/T98834#1278678 (10mmodell) [16:41:31] (03CR) 10Paladox: "This adds the tests the extension needs when it starts being used." [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T106002) (owner: 10Paladox) [16:44:58] (03CR) 10BryanDavis: "This *almost* works. When testing in beta cluster I have found that there are a number of files in /srv/mediawiki-staging (mostly symlinks" [tools/scap] - 10https://gerrit.wikimedia.org/r/224313 (https://phabricator.wikimedia.org/T104826) (owner: 10BryanDavis) [16:45:25] 10Beta-Cluster, 10MediaWiki-extensions-CentralAuth: Central login at betawiki doesn't work - https://phabricator.wikimedia.org/T106035#1457760 (10Krenair) It works for me. Have you tried clearing your browser's cookies? Sometimes a forceHTTPS cookie gets set that causes issues like this. [16:47:22] 10Beta-Cluster, 10Deployment-Systems, 7I18n: Beta dewiki's strings are in English when logged in (does not happen in debug mode or other languages) - https://phabricator.wikimedia.org/T106019#1457774 (10Krenair) Works for me? [16:49:32] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1457782 (10Krinkle) Using Git information for this is imho infeasible, and needlessly complex. Whethe... [16:49:45] 10Deployment-Systems, 6Release-Engineering, 7Epic: Rethinking mediawiki deployment branches and release process - https://phabricator.wikimedia.org/T89945#1457787 (10mmodell) In T104398#1456723 I elaborate a bit about how the long-lived branches might be managed. [16:52:29] (03PS3) 10Paladox: Add tests for WikibaseRepository [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T75863) [16:54:11] (03PS4) 10Paladox: Add tests for WikibaseRepository [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T106002) [16:54:30] (03PS5) 10Paladox: Add tests for WikibaseRepository [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T75863) [17:05:13] 10Beta-Cluster, 10MediaWiki-extensions-CentralAuth: Central login at betawiki doesn't work - https://phabricator.wikimedia.org/T106035#1457860 (10MGChecker) 5Open>3Resolved a:3MGChecker Thank you, after deleting all cookies of wmflabs it works. [17:25:16] PROBLEM - Puppet failure on nodepool-t105406 is CRITICAL 100.00% of data above the critical threshold [0.0] [17:46:52] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1458063 (10BBlack) >>! In T99096#1457782, @Krinkle wrote: > Using Git information for this is imho in... [18:42:24] (03CR) 10JanZerebecki: [C: 04-1] "This should not be done until the content of the repos is there, which is intended to be split off from Wikibase." [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T75863) (owner: 10Paladox) [18:53:29] Project UploadWizard-api-commons.wikimedia.org build #1782: FAILURE in 28 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.org/1782/ [18:57:23] (03CR) 10Paladox: "Ok sorry." [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T75863) (owner: 10Paladox) [18:59:52] RECOVERY - Puppet failure on mira is OK Less than 1.00% above the threshold [0.0] [18:59:57] (03CR) 10Paladox: "Hi could you copy it over to this repo please." [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T75863) (owner: 10Paladox) [19:20:30] (03PS1) 10Paladox: Update CheckUser tests [integration/config] - 10https://gerrit.wikimedia.org/r/225182 [19:20:41] (03PS2) 10Paladox: Update CheckUser tests [integration/config] - 10https://gerrit.wikimedia.org/r/225182 [19:21:05] (03CR) 10JanZerebecki: "The blockers of https://phabricator.wikimedia.org/T75863 and probably more need to be solved first." [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T75863) (owner: 10Paladox) [19:22:01] (03PS3) 10Paladox: Update CheckUser tests [integration/config] - 10https://gerrit.wikimedia.org/r/225182 [19:22:44] (03CR) 10Paladox: "OK." [integration/config] - 10https://gerrit.wikimedia.org/r/225086 (https://phabricator.wikimedia.org/T75863) (owner: 10Paladox) [20:07:53] 10Beta-Cluster, 10Deployment-Systems, 7I18n: Beta dewiki's strings are in English when logged in (does not happen in debug mode or other languages) - https://phabricator.wikimedia.org/T106019#1458616 (10Luke081515) At the moment, it seems to be okay, was there a change? [20:31:47] (03PS1) 10Paladox: Remove unused mwext-MaintenanceShell test [integration/config] - 10https://gerrit.wikimedia.org/r/225220 [20:32:08] (03PS2) 10Paladox: Remove unused mwext-MaintenanceShell test [integration/config] - 10https://gerrit.wikimedia.org/r/225220 [20:35:45] (03PS1) 10Paladox: Update Maintenance extension tests [integration/config] - 10https://gerrit.wikimedia.org/r/225222 [20:38:07] 10Beta-Cluster, 10Deployment-Systems, 7I18n: Beta dewiki's strings are in English when logged in (does not happen in debug mode or other languages) - https://phabricator.wikimedia.org/T106019#1458741 (10greg) It could have been a bad deploy of code to the beta cluster: https://integration.wikimedia.org/ci/vi... [20:42:45] (03PS2) 10Paladox: Update Maintenance extension tests [integration/config] - 10https://gerrit.wikimedia.org/r/225222 [20:43:20] (03PS3) 10Paladox: Update Maintenance extension tests [integration/config] - 10https://gerrit.wikimedia.org/r/225222 [20:45:03] (03PS11) 10Paladox: Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 [20:45:12] (03CR) 10Paladox: [C: 031] Use npm for CollapsibleVector [integration/config] - 10https://gerrit.wikimedia.org/r/224819 (owner: 10Paladox) [20:50:45] (03PS1) 10Paladox: Update MsUpload tests [integration/config] - 10https://gerrit.wikimedia.org/r/225231 [20:53:24] 10Beta-Cluster, 10MediaWiki-extensions-WikimediaMaintenance: Check/update dumpInterwiki for beta usage - https://phabricator.wikimedia.org/T75906#1458782 (10Krenair) a:3Krenair [20:56:56] (03PS2) 10Paladox: Update MsUpload tests [integration/config] - 10https://gerrit.wikimedia.org/r/225231 [21:01:07] (03PS4) 10Thcipriani: Add service deploy via scap [tools/scap] - 10https://gerrit.wikimedia.org/r/224374 [21:09:46] (03PS1) 10Paladox: Update MwEmbedSupport tests [integration/config] - 10https://gerrit.wikimedia.org/r/225239 [21:12:50] (03PS3) 10Paladox: Update MsUpload tests [integration/config] - 10https://gerrit.wikimedia.org/r/225231 [21:21:40] (03CR) 10Luis Felipe Schenone: [C: 031] Update MsUpload tests [integration/config] - 10https://gerrit.wikimedia.org/r/225231 (owner: 10Paladox) [21:22:40] (03PS2) 10Paladox: Update MwEmbedSupport tests [integration/config] - 10https://gerrit.wikimedia.org/r/225239 [21:24:43] (03PS3) 10Paladox: Update MwEmbedSupport tests [integration/config] - 10https://gerrit.wikimedia.org/r/225239 [21:33:54] 10Deployment-Systems, 10Traffic, 6operations: Varnish cache busting desired for /static/$VERSION/ resources which change within the lifetime of a WMF release branch - https://phabricator.wikimedia.org/T99096#1458885 (10Krinkle) >>! In T99096#1458063, @BBlack wrote: >>>! In T99096#1457782, @Krinkle wrote: >>... [21:36:12] (03CR) 10Hashar: "Bah took 2 hours to run on labs https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-2/5/console :D" [integration/config] - 10https://gerrit.wikimedia.org/r/225063 (https://phabricator.wikimedia.org/T93559) (owner: 10Hashar) [21:40:28] (03PS1) 10Paladox: Update TimedMediaHandler tests [integration/config] - 10https://gerrit.wikimedia.org/r/225246 [21:40:53] 10Browser-Tests: Write the first browsertests/Selenium test - https://phabricator.wikimedia.org/T94024#1458892 (10zeljkofilipin) [21:40:55] (03PS2) 10Paladox: Update TimedMediaHandler tests [integration/config] - 10https://gerrit.wikimedia.org/r/225246 [21:40:56] 10Browser-Tests: Fix broken browsertests/Selenium Jenkins jobs - https://phabricator.wikimedia.org/T94299#1458895 (10zeljkofilipin) [21:42:28] 10Browser-Tests, 10Wikimania-Hackathon-2015, 7I18n: Hacking: Load i18n messages from MediaWiki to browser tests - https://phabricator.wikimedia.org/T90577#1458901 (10zeljkofilipin) a:5zeljkofilipin>3Vikassy [21:43:01] 10Browser-Tests, 10Wikimania-Hackathon-2015, 7I18n: Hacking: Load i18n messages from MediaWiki to browser tests - https://phabricator.wikimedia.org/T90577#1062230 (10zeljkofilipin) @vikassy and I paired on this today. @vikassy, please push the patch to gerrit. [21:43:11] 10Browser-Tests: Investigate using the sikuli-like Applitools framework for visual testing - https://phabricator.wikimedia.org/T90884#1458914 (10zeljkofilipin) [21:43:46] 10Browser-Tests: Write the first browsertests/Selenium test - https://phabricator.wikimedia.org/T94024#1458918 (10zeljkofilipin) a:5zeljkofilipin>3None [21:43:53] 10Browser-Tests: Fix broken browsertests/Selenium Jenkins jobs - https://phabricator.wikimedia.org/T94299#1458919 (10zeljkofilipin) a:5zeljkofilipin>3None [21:44:58] 10Browser-Tests: Investigate using the sikuli-like Applitools framework for visual testing - https://phabricator.wikimedia.org/T90884#1458923 (10zeljkofilipin) a:5zeljkofilipin>3None [21:47:10] 6Release-Engineering, 7Tracking: Update repositories that use mediawiki_selenium Ruby gem 1.x (tracking) - https://phabricator.wikimedia.org/T94083#1458927 (10zeljkofilipin) [21:50:50] (03PS9) 10Paladox: Update tests for Metrolook [integration/config] - 10https://gerrit.wikimedia.org/r/224834 [21:51:07] (03PS10) 10Paladox: Update tests for Vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [21:51:13] (03PS11) 10Paladox: Update tests for Vector skin [integration/config] - 10https://gerrit.wikimedia.org/r/224824 [23:31:44] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0]