[00:55:38] PROBLEM - Puppet staleness on integration-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [01:46:39] 10Browser-Tests, 10VisualEditor, 5WMF-deploy-2015-10-13_(1.27.0-wmf.3): No such file or directory - i18n/af.json (Errno::ENOENT) - https://phabricator.wikimedia.org/T111525#1733722 (10Jdforrester-WMF) p:5Normal>3Unbreak! [01:47:12] 10Browser-Tests, 10VisualEditor, 5WMF-deploy-2015-10-13_(1.27.0-wmf.3): No such file or directory - i18n/af.json (Errno::ENOENT) - https://phabricator.wikimedia.org/T111525#1606436 (10Jdforrester-WMF) This change has broken every single browser test on all platforms now for a week. Could you please fix it or... [03:15:52] PROBLEM - Host deployment-mediawiki01 is DOWN: PING CRITICAL - Packet loss = 80%, RTA = 2214.00 ms [03:16:08] RECOVERY - Host deployment-mediawiki01 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms [03:16:41] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #753: 04FAILURE in 26 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/753/ [05:26:07] PROBLEM - Puppet failure on integration-slave-trusty-1017 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [05:26:12] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #576: 04FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/576/ [06:06:12] RECOVERY - Puppet failure on integration-slave-trusty-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [06:37:25] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [08:32:37] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #756: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/756/ [08:49:48] (03PS2) 10Hashar: [EventLogging] Update tox tests [integration/config] - 10https://gerrit.wikimedia.org/r/246845 (owner: 10Paladox) [08:50:19] (03PS3) 10Hashar: [EventLogging] switch to tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/246845 (owner: 10Paladox) [08:51:25] (03CR) 10Hashar: [C: 032] [EventLogging] switch to tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/246845 (owner: 10Paladox) [08:53:30] (03Merged) 10jenkins-bot: [EventLogging] switch to tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/246845 (owner: 10Paladox) [09:01:41] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:35:39] (03Abandoned) 10Hashar: Add new template flake8-py27 [integration/config] - 10https://gerrit.wikimedia.org/r/246776 (owner: 10Paladox) [09:36:31] (03CR) 10Hashar: [C: 04-1] "Pending source change https://gerrit.wikimedia.org/r/#/c/246841/" [integration/config] - 10https://gerrit.wikimedia.org/r/246773 (https://phabricator.wikimedia.org/T88343) (owner: 10Paladox) [09:36:36] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:38:26] 10Beta-Cluster-Infrastructure, 10ContentTranslation-Deployments, 10Wikimedia-Site-Requests, 5Patch-For-Review: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1734261 (10hashar) 5Resolved>3Open Please reopen the eswiki on beta cluster. It is used for content translation te... [09:43:20] 10Beta-Cluster-Infrastructure, 10ContentTranslation-Deployments, 10Wikimedia-Site-Requests, 5Patch-For-Review: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1734270 (10Luke081515) a:5Krenair>3Luke081515 [09:50:18] 10MediaWiki-Codesniffer, 10Possible-Tech-Projects, 3Outreachy-Round-11: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1734281 (1001tonythomas) I am shifting this to #Outreachy-11 as the project description has two mentors, micro-tasks and looks ready for the 11th ed... [09:52:04] 6Release-Engineering-Team, 6operations: Proposal: Reroute all requests from WMF Office IPs to a machine/machines similar to the canary cluster - https://phabricator.wikimedia.org/T115631#1734290 (10hashar) If I remember correctly, we already discussed this back in January 2015 (on the ops list?) and rejected i... [09:52:19] 10Beta-Cluster-Infrastructure, 10ContentTranslation-Deployments, 10Wikimedia-Site-Requests, 5Patch-For-Review: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1734293 (10Peachey88) >>! In T109157#1734261, @hashar wrote: > Please reopen the eswiki on beta cluster. It is used fo... [09:53:18] 10Beta-Cluster-Infrastructure, 10ContentTranslation-Deployments, 10Wikimedia-Site-Requests, 5Patch-For-Review: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1734302 (10Luke081515) Please note: It is possible to write at eswiki at the moment, till T115584 is fixed, but I woul... [09:54:26] 10Beta-Cluster-Infrastructure, 10ContentTranslation-Deployments, 10Wikimedia-Site-Requests, 5Patch-For-Review: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1734311 (10hashar) 5Open>3Resolved > Hello, beta eswiki was used for the ContentTranslation Project, but there was... [09:57:16] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/246845 (owner: 10Paladox) [10:00:02] 10Beta-Cluster-Infrastructure, 10ContentTranslation-Deployments, 10Wikimedia-Site-Requests, 5Patch-For-Review: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157#1734325 (10Luke081515) @Peachey88: simplewiki, enwiki (these are the most actives). But there are others too: http://d... [10:53:30] (03PS4) 10Hashar: [SacredText] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/245967 (owner: 10Paladox) [10:53:39] (03CR) 10Hashar: [C: 04-1] "Depends on https://gerrit.wikimedia.org/r/#/c/239173/" [integration/config] - 10https://gerrit.wikimedia.org/r/245967 (owner: 10Paladox) [10:55:14] (03PS2) 10Hashar: [RelatedArticles] Re add composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/246984 (owner: 10Paladox) [10:55:43] (03CR) 10Hashar: [C: 032] [RelatedArticles] Re add composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/246984 (owner: 10Paladox) [10:56:53] (03Merged) 10jenkins-bot: [RelatedArticles] Re add composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/246984 (owner: 10Paladox) [13:05:31] (03PS1) 10Hashar: [Elastica] Migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247264 [13:07:19] (03CR) 10jenkins-bot: [V: 04-1] [Elastica] Migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247264 (owner: 10Hashar) [13:12:29] (03PS2) 10Hashar: [Elastica] Migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247264 [13:18:06] !log Adding Elastica to the extensions gate ( https://gerrit.wikimedia.org/r/#/c/247264/ ) [13:18:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:18:54] (03CR) 10Hashar: [C: 032] "Jobs updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/247264 (owner: 10Hashar) [13:20:47] (03Merged) 10jenkins-bot: [Elastica] Migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247264 (owner: 10Hashar) [13:25:29] (03CR) 10Hashar: "Validated via https://gerrit.wikimedia.org/r/#/c/145536/" [integration/config] - 10https://gerrit.wikimedia.org/r/247264 (owner: 10Hashar) [13:36:11] (03PS1) 10Hashar: [Cite] Migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247266 [13:37:15] (03CR) 10Hashar: [C: 032] "Jobs updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/247266 (owner: 10Hashar) [13:39:08] (03Merged) 10jenkins-bot: [Cite] Migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247266 (owner: 10Hashar) [13:40:27] !log Adding Cite to the extensions gate ( https://gerrit.wikimedia.org/r/#/c/247266/ ) [13:40:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:50:11] (03CR) 10Hashar: "Verified on https://gerrit.wikimedia.org/r/#/c/69653/" [integration/config] - 10https://gerrit.wikimedia.org/r/247266 (owner: 10Hashar) [13:56:04] (03PS1) 10Hashar: [MwEmbedSupport] Migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247271 [14:01:06] (03CR) 10Hashar: [C: 032] "Jobs updated" [integration/config] - 10https://gerrit.wikimedia.org/r/247271 (owner: 10Hashar) [14:03:27] (03Merged) 10jenkins-bot: [MwEmbedSupport] Migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247271 (owner: 10Hashar) [14:05:27] !log Adding MwEmbedSupport to the extensions gate ( https://gerrit.wikimedia.org/r/#/c/247271/ ) [14:05:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:06:43] (03CR) 10Hashar: "Validated on https://gerrit.wikimedia.org/r/#/c/136049/" [integration/config] - 10https://gerrit.wikimedia.org/r/247271 (owner: 10Hashar) [14:12:40] (03PS1) 10Hashar: [TimedMediaHandler] Migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247273 [14:17:00] (03CR) 10Hashar: [C: 032] "Jobs updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/247273 (owner: 10Hashar) [14:18:42] (03Merged) 10jenkins-bot: [TimedMediaHandler] Migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247273 (owner: 10Hashar) [14:21:55] (03CR) 10Hashar: "Validated on https://gerrit.wikimedia.org/r/#/c/36974/" [integration/config] - 10https://gerrit.wikimedia.org/r/247273 (owner: 10Hashar) [14:26:57] !log Adding TimedMediaHandler to the extensions gate ( https://gerrit.wikimedia.org/r/#/c/247273/ ) [14:27:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:37:44] (03PS1) 10Hashar: [PdfHandler] migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247278 [14:38:28] !log Adding PdfHandler to the extensions gate ( https://gerrit.wikimedia.org/r/#/c/247278/ ) [14:38:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:39:25] (03CR) 10Hashar: [C: 032] [PdfHandler] migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247278 (owner: 10Hashar) [14:40:34] (03Merged) 10jenkins-bot: [PdfHandler] migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247278 (owner: 10Hashar) [14:44:08] (03CR) 10Hashar: "Validated on https://gerrit.wikimedia.org/r/#/c/136065/" [integration/config] - 10https://gerrit.wikimedia.org/r/247278 (owner: 10Hashar) [14:45:03] (03CR) 10Hashar: "I have migrated all the dependencies to be run in the testextensions jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/246248 (owner: 10Mattflaschen) [14:46:08] (03CR) 10Hashar: "Still have to migrate CirrusSearch... Will do later today." [integration/config] - 10https://gerrit.wikimedia.org/r/246248 (owner: 10Mattflaschen) [14:51:09] (03PS1) 10Hashar: [CirrusSearch] migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247280 [14:51:36] !log Adding CirrusSearch to the extensions gate ( https://gerrit.wikimedia.org/r/#/c/247280/ ) [14:51:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:53:14] (03CR) 10jenkins-bot: [V: 04-1] [CirrusSearch] migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247280 (owner: 10Hashar) [14:53:21] hashar or anyone: Any idea what's going on with the unit test failure on https://gerrit.wikimedia.org/r/#/c/247268/? [14:54:08] (03PS2) 10Hashar: [CirrusSearch] migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247280 [14:54:56] anomie: ooops [14:55:33] anomie: I have been added a few more extensions to those jobs [14:55:43] anomie: the tests pass on master branches [14:55:46] but apparently fails there :/ [14:56:12] seems on wmf/1.27.0-wmf.2 Flow invokes something which is not available in Elastica ? :( [14:56:45] (03CR) 10Hashar: [C: 032] [CirrusSearch] migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247280 (owner: 10Hashar) [14:56:51] Looks like it was broken in e759d9601f0c0a54ade74fe844970fb6af605c37 but somehow no unit tests caught it at the time? The fix is f2d613b9a3255a204d0ef05e74cedb86efca7af3, but that's not in wmf.3. [14:57:12] oh you are fast [14:57:42] anomie: mind cherry picking it to the wmf branch ? [14:58:33] (03Merged) 10jenkins-bot: [CirrusSearch] migrate to extensions gate [integration/config] - 10https://gerrit.wikimedia.org/r/247280 (owner: 10Hashar) [14:59:05] I can cherry-pick it as long as nothing else breaks. [15:00:26] hopefully it is just a corner case [15:00:28] :-/ [15:02:26] (03CR) 10Hashar: "Validated on https://gerrit.wikimedia.org/r/#/c/70205/" [integration/config] - 10https://gerrit.wikimedia.org/r/247280 (owner: 10Hashar) [15:02:53] (03CR) 10Hashar: "I have added CirrusSearch with https://gerrit.wikimedia.org/r/#/c/247280/" [integration/config] - 10https://gerrit.wikimedia.org/r/246248 (owner: 10Mattflaschen) [15:07:00] 6Release-Engineering-Team, 6operations: Proposal: Reroute all requests from WMF Office IPs to a machine/machines similar to the canary cluster - https://phabricator.wikimedia.org/T115631#1734987 (10faidon) The sets of "people working on the code and/or are willing to do QA" and "people who are connected to the... [15:08:09] anomie: congratulations, the job pass -:) [15:10:58] (03CR) 10JanZerebecki: "Dependency is merged." [integration/config] - 10https://gerrit.wikimedia.org/r/246819 (owner: 10JanZerebecki) [15:11:03] (03PS4) 10JanZerebecki: Add repo wikidata/query/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/246819 [15:12:58] (03CR) 10JanZerebecki: [C: 032] Add repo wikidata/query/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/246819 (owner: 10JanZerebecki) [15:16:09] out going to grab my daughter [15:20:29] (03CR) 10JanZerebecki: [C: 04-1] "This is inconsistent with what the bug says. Can we clear up first what the intention is here?" [integration/config] - 10https://gerrit.wikimedia.org/r/246712 (https://phabricator.wikimedia.org/T115061) (owner: 10Paladox) [15:21:40] 6Release-Engineering-Team, 6operations, 15User-greg: Proposal: Reroute all requests from WMF Office IPs to a machine/machines similar to the canary cluster - https://phabricator.wikimedia.org/T115631#1735031 (10greg) 5Open>3declined a:3greg Declining, not worth the time. [15:25:07] (03Merged) 10jenkins-bot: Add repo wikidata/query/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/246819 (owner: 10JanZerebecki) [15:25:56] (03CR) 10JanZerebecki: [C: 032] Delete failing ContentTranslation browsertests Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/245469 (https://phabricator.wikimedia.org/T113734) (owner: 10Zfilipin) [15:26:29] (03PS2) 10JanZerebecki: Delete failing ContentTranslation browsertests Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/245469 (https://phabricator.wikimedia.org/T113734) (owner: 10Zfilipin) [15:26:35] (03CR) 10JanZerebecki: [C: 032] Delete failing ContentTranslation browsertests Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/245469 (https://phabricator.wikimedia.org/T113734) (owner: 10Zfilipin) [15:29:51] (03Merged) 10jenkins-bot: Delete failing ContentTranslation browsertests Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/245469 (https://phabricator.wikimedia.org/T113734) (owner: 10Zfilipin) [15:33:58] (03PS2) 10JanZerebecki: Add experimental rake-jessie to a bunch of repos [integration/config] - 10https://gerrit.wikimedia.org/r/246268 (https://phabricator.wikimedia.org/T114860) (owner: 10Hashar) [15:34:39] (03CR) 10JanZerebecki: "PS2: is an automatic rebase with kdiff3." [integration/config] - 10https://gerrit.wikimedia.org/r/246268 (https://phabricator.wikimedia.org/T114860) (owner: 10Hashar) [15:36:41] (03CR) 10JanZerebecki: [C: 032] Add experimental rake-jessie to a bunch of repos [integration/config] - 10https://gerrit.wikimedia.org/r/246268 (https://phabricator.wikimedia.org/T114860) (owner: 10Hashar) [15:38:30] (03Merged) 10jenkins-bot: Add experimental rake-jessie to a bunch of repos [integration/config] - 10https://gerrit.wikimedia.org/r/246268 (https://phabricator.wikimedia.org/T114860) (owner: 10Hashar) [15:38:51] (03CR) 10JanZerebecki: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/228580 (owner: 10Awight) [15:38:53] (03CR) 10jenkins-bot: [V: 04-1] WIP More tests for SmashPig [integration/config] - 10https://gerrit.wikimedia.org/r/228580 (owner: 10Awight) [15:43:50] (03CR) 10JanZerebecki: [C: 04-1] "Maybe I missunderstood the intent of this, but maybe you want to change zuul/layout.yaml to add to the repo of your choosing:" [integration/config] - 10https://gerrit.wikimedia.org/r/228580 (owner: 10Awight) [15:44:52] (03CR) 10JanZerebecki: "Btw.: JJB is run by the CI, you can look at integration-jjb-config-diff to see your changes and integration-zuul-layoutdiff for zuul." [integration/config] - 10https://gerrit.wikimedia.org/r/228580 (owner: 10Awight) [15:45:18] (03PS12) 10JanZerebecki: Update BlueSky tests [integration/config] - 10https://gerrit.wikimedia.org/r/226635 (owner: 10Paladox) [15:45:28] (03CR) 10JanZerebecki: [C: 032] Update BlueSky tests [integration/config] - 10https://gerrit.wikimedia.org/r/226635 (owner: 10Paladox) [15:46:30] (03Merged) 10jenkins-bot: Update BlueSky tests [integration/config] - 10https://gerrit.wikimedia.org/r/226635 (owner: 10Paladox) [15:46:33] (03CR) 10JanZerebecki: [C: 04-1] "Per Krinkle." [integration/config] - 10https://gerrit.wikimedia.org/r/242939 (owner: 10Krinkle) [15:49:10] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/226635 (owner: 10Paladox) [15:49:56] (03PS3) 10JanZerebecki: Delete failing VisualEditor browsertests Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/245499 (https://phabricator.wikimedia.org/T94162) (owner: 10Zfilipin) [15:50:03] (03CR) 10JanZerebecki: [C: 032] Delete failing VisualEditor browsertests Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/245499 (https://phabricator.wikimedia.org/T94162) (owner: 10Zfilipin) [15:51:00] (03PS2) 10Paladox: [WebPlatformAuth] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/246712 (https://phabricator.wikimedia.org/T115061) [15:51:55] (03Merged) 10jenkins-bot: Delete failing VisualEditor browsertests Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/245499 (https://phabricator.wikimedia.org/T94162) (owner: 10Zfilipin) [15:52:12] 10Browser-Tests, 10VisualEditor, 5Patch-For-Review: Delete or fix failed VisualEditor browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94162#1735110 (10JanZerebecki) Jobs still need to be manually deleted: 16:12:47 Only in output-parent: browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-l... [15:55:15] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #607: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/607/ [16:03:15] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:03:15] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:03:46] 10Browser-Tests, 10VisualEditor, 5Patch-For-Review: Delete or fix failed VisualEditor browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94162#1735127 (10zeljkofilipin) >>! In T94162#1735110, @JanZerebecki wrote: > Jobs still need to be manually deleted: > 16:12:47 Only in output-parent: browsert... [16:10:46] (03PS1) 10JanZerebecki: Replace some pywikibot/core jobs with the generic tox-jessie version. [integration/config] - 10https://gerrit.wikimedia.org/r/247291 (https://phabricator.wikimedia.org/T87169) [16:23:49] 10Browser-Tests, 10VisualEditor: Delete or fix failed VisualEditor browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94162#1735197 (10zeljkofilipin) a:5zeljkofilipin>3Amire80 [16:24:44] 10Browser-Tests, 5Release-Engineering-Epics, 7Epic, 7Tracking: Fix or delete failing browser tests Jenkins jobs - https://phabricator.wikimedia.org/T94150#1735200 (10zeljkofilipin) [16:24:46] 10Browser-Tests, 10MediaWiki-extensions-ContentTranslation, 5Patch-For-Review, 7WorkType-Maintenance: Delete failing ContentTranslation browsertests Jenkins jobs - https://phabricator.wikimedia.org/T113734#1735199 (10zeljkofilipin) 5Open>3Resolved [16:28:25] 10Browser-Tests, 10VisualEditor, 5WMF-deploy-2015-10-13_(1.27.0-wmf.3): No such file or directory - i18n/af.json (Errno::ENOENT) - https://phabricator.wikimedia.org/T111525#1735212 (10zeljkofilipin) @Vikassy while resolving T94162 I have deleted all VisualEditor jobs except one (that does not have this error... [16:30:17] James_F: greg-g says VE team is running browser tests locally and https://phabricator.wikimedia.org/T111525 is bugging you [16:30:19] did I understand him correctly? [16:30:27] zeljkof: Yes. [16:31:11] zeljkof: But you just deleted them instead of fixing the bug you introduced when you merged https://gerrit.wikimedia.org/r/#/c/241248/ :-( [16:34:40] James_F: I can work on that tomorrow if it is urgent for you, and if vikas does not have the time [16:35:06] James_F: well, the jobs were broken forever and nobody commented on the bug, I thought nobody cared [16:35:14] it is easy to recreate the jobs [16:39:34] vikassy: yo! [16:39:52] James_F says https://phabricator.wikimedia.org/T111525 makes him sad :( [16:40:12] it would be great if you could take a look, or else I can work on it tomorrow [16:40:16] just let me know [16:40:22] (03CR) 10JanZerebecki: [C: 032] Replace some pywikibot/core jobs with the generic tox-jessie version. [integration/config] - 10https://gerrit.wikimedia.org/r/247291 (https://phabricator.wikimedia.org/T87169) (owner: 10JanZerebecki) [16:40:39] I can recreate job(s) if you can not reproduce the problem locally [16:42:17] (03Merged) 10jenkins-bot: Replace some pywikibot/core jobs with the generic tox-jessie version. [integration/config] - 10https://gerrit.wikimedia.org/r/247291 (https://phabricator.wikimedia.org/T87169) (owner: 10JanZerebecki) [16:42:33] zeljkof: Hola ! [16:42:35] vikassy: also https://phabricator.wikimedia.org/T94162 needs some love ;) [16:43:02] zeljkof: James_F: Yup ! Fixing it .. [16:43:12] vikassy: thanks! [16:43:22] wee [16:46:27] !log reloading zuul for c6ca369..d9cae0a [16:46:35] James_F: if you would like to talk about browser tests, I am all ears [16:47:17] if you want jobs recreated, or if you need help with the tests, I will be glad to help [16:47:57] I just get really really sad when a job is failing for a month :'( I can not watch is suffer, I have to help it get to a better place [16:49:00] (03CR) 10JanZerebecki: "*sigh* forgot to fix the commit message. It should be: add npm to wikidata/query/rdf" [integration/config] - 10https://gerrit.wikimedia.org/r/246819 (owner: 10JanZerebecki) [16:52:59] 10Continuous-Integration-Infrastructure, 10pywikibot-core, 5Patch-For-Review: run at least pep8 and pep257 for new changesets submitted to pywikibot/core for any user - https://phabricator.wikimedia.org/T87169#1735321 (10JanZerebecki) @Hashar Can we add the tox-jessie job also to the check pipeline to make i... [17:13:32] (03PS1) 10JanZerebecki: Move jshint to experimental on mediawiki/skins/BlueSky as it is failing [integration/config] - 10https://gerrit.wikimedia.org/r/247302 [17:15:13] (03CR) 10JanZerebecki: [C: 032] Move jshint to experimental on mediawiki/skins/BlueSky as it is failing [integration/config] - 10https://gerrit.wikimedia.org/r/247302 (owner: 10JanZerebecki) [17:16:15] (03Merged) 10jenkins-bot: Move jshint to experimental on mediawiki/skins/BlueSky as it is failing [integration/config] - 10https://gerrit.wikimedia.org/r/247302 (owner: 10JanZerebecki) [17:17:44] !log reloading zuul for d9cae0a..9362473 [17:22:23] 10Browser-Tests, 10VisualEditor, 5WMF-deploy-2015-10-13_(1.27.0-wmf.3): No such file or directory - i18n/af.json (Errno::ENOENT) - https://phabricator.wikimedia.org/T111525#1735524 (10Vikassy) These 2 are different issue. Pointing to the correct issue: T90577 (this is where the problem is) [17:29:34] weird ! Not able to send code to review from office internet. Let me use other connection and try it .. [17:38:34] zeljkof: Fixed it in my local machine. I will push it when I get back to my home. Not able to push from my office :( [17:39:14] vikassy: great! thanks [17:43:28] (03Abandoned) 10Mattflaschen: Add CirrusSearch and Elastica as Flow dependencies, sort [integration/config] - 10https://gerrit.wikimedia.org/r/246248 (owner: 10Mattflaschen) [17:46:46] hasharAway: Please merge https://gerrit.wikimedia.org/r/#/c/226634/ adds npm and fixes js issues. [17:47:33] jzerebecki: I fixed js issues here https://gerrit.wikimedia.org/r/#/c/226634/ [17:48:15] paladox: i don't have merge rights there [17:48:25] Oh ok. [17:51:21] 6Release-Engineering-Team, 3Scap3, 6operations: Move scap target configuration to etcd - https://phabricator.wikimedia.org/T115899#1735644 (10demon) 3NEW [17:52:21] (03PS1) 10Paladox: [BlueSky] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/247309 [17:53:16] (03CR) 10jenkins-bot: [V: 04-1] [BlueSky] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/247309 (owner: 10Paladox) [17:53:26] (03PS2) 10Paladox: [BlueSky] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/247309 [17:54:42] (03PS3) 10Paladox: [BlueSky] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/247309 [17:56:09] 10Deployment-Systems, 3Scap3: Scap3 should have idempotent deploys - https://phabricator.wikimedia.org/T109513#1735679 (10dduvall) a:3thcipriani [17:56:12] 6Release-Engineering-Team, 3Scap3, 6operations: Move scap target configuration to etcd - https://phabricator.wikimedia.org/T115899#1735681 (10Krenair) Do we get anything useful out of this? Or is this task just because it's the cool thing to do? [17:56:18] 10Deployment-Systems, 3Scap3: Scap3 should have idempotent deploys - https://phabricator.wikimedia.org/T109513#1550785 (10dduvall) p:5Triage>3Normal [17:57:49] 6Release-Engineering-Team, 3Scap3, 6operations: Move scap target configuration to etcd - https://phabricator.wikimedia.org/T115899#1735689 (10Dzahn) We get to kill the entire dsh module (remove cruft) which is not used anymore except for this. [17:58:22] 6Release-Engineering-Team, 3Scap3, 6operations: Move scap target configuration to etcd - https://phabricator.wikimedia.org/T115899#1735693 (10Dzahn) also, it's kind of a way to resolve T80395 [18:00:17] 6Release-Engineering-Team, 3Scap3, 6operations: Move scap target configuration to etcd - https://phabricator.wikimedia.org/T115899#1735706 (10Dzahn) [18:01:42] 6Release-Engineering-Team, 3Scap3, 6operations: Move scap target configuration to etcd - https://phabricator.wikimedia.org/T115899#1735711 (10demon) >>! In T115899#1735681, @Krenair wrote: > Do we get anything useful out of this? Or is this task just because it's the cool thing to do? If we've already got t... [18:15:43] thcipriani, marxarelli, twentyafterfour: ^ [18:15:51] That may or may not be a good idea, thoughts welcome [19:01:43] 10Continuous-Integration-Config: Move Bundler Jenkins jobs to Nodepool instances - https://phabricator.wikimedia.org/T114860#1735947 (10hashar) [19:08:01] Is it possible to make so that any change in gerrit against wikidata/query/rdf would add me as a reviewer? [19:13:33] SMalyshev: Yes, there's a bot that can do it. [19:13:36] Lemme find the page... [19:14:21] SMalyshev: https://www.mediawiki.org/wiki/Git/Reviewers is what you want [19:15:32] I just use https://gerrit.wikimedia.org/r/#/settings/projects [19:15:39] ostriches: thanks! [19:15:54] yw [19:15:57] bd808: That also works [19:16:33] SMalyshev: bd808: the difference between both option is Gerrit watching system spam you notification for repositories and does not add you as a reviewer. [19:17:10] SMalyshev: bd808 the Git/Reviewers bot add you as a reviewer (and thus you are subscribed to mail spam), but it also comes with a filter to only subscribe you to specific changes [19:17:12] so [19:17:23] you could subscribe to any repositories changes to /composer.json [19:17:45] cool [19:18:18] I am added as a reviewer to anything under `integration/*` [19:22:28] SMalyshev: There's also an entry for Cirrus that you might want to add yourself to as well [19:22:49] ostriches: true, thanks [19:29:01] I hate systemctl [19:29:49] 10Beta-Cluster-Infrastructure, 7Puppet: Puppet failures across all beta caches due to *.wmflabs.org certificate - https://phabricator.wikimedia.org/T115238#1736015 (10thcipriani) [19:32:35] 10Beta-Cluster-Infrastructure: closed-labs.dblist no longer read as 'closed' tag - https://phabricator.wikimedia.org/T115584#1736025 (10mmodell) p:5Triage>3Normal [19:34:12] 10Beta-Cluster-Infrastructure: closed-labs.dblist no longer read as 'closed' tag - https://phabricator.wikimedia.org/T115584#1727961 (10mmodell) I don't understand the reasoning for 7eda03ada4c5ec0c3bb3b2969b2ab07935e5bb29, the whole point of realm specific stuff is that we have multiple realms and they matter.... [19:34:58] 10Beta-Cluster-Infrastructure, 6Language-Engineering, 7I18n: l10nupdate key not set up properly - https://phabricator.wikimedia.org/T114971#1736032 (10mmodell) p:5Triage>3Normal [19:39:24] ostriches: thcipriani haven't you tried to setup Swift on the staging cluster ? [19:40:09] I started poking it, it wasn't done though. [19:40:12] I am starting to wonder how complicated it is to setup a tiny Swift cluster [19:40:18] It's not too hard. [19:40:28] can it be used standalone ? [19:40:43] ie without the rest of the OpenStack environment (i.e. Keystone ) [19:40:56] Yes [19:41:24] CI has ton of uses for an object stores :D [19:41:32] guess I will want to write down something somewhere [19:43:37] hashar: fwiw, here's the ticket for staging where most of the work was tracked for Swift: https://phabricator.wikimedia.org/T91553 [19:43:58] nice ! [19:46:18] thcipriani: ostriches the CI counterpart is https://phabricator.wikimedia.org/T114998 [19:50:15] hashar: Fwiw, we've mostly moved to jessie caches. The only place we're stuck is the usual ssl stupidity [19:50:35] ie: it's all running, but puppet barfs. [19:50:44] well [19:50:54] maybe we can remove the protoproxy stuff on beta ? [19:51:04] i.e. get rid of nginx and all the related stuff including the ssl certs [19:51:06] the thing is [19:51:09] it used to work [19:51:30] we had some *.wmflabs.org self signed cert there [19:51:32] SSL_CTX_use_PrivateKey_file("/etc/ssl/private/star.wmflabs.org.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch) [19:51:45] We have the cert, it's just barfing on it methinks. [19:52:58] $ find .|grep star.wmf [19:52:59] ./files/ssl/star.wmflabs.key [19:53:00] ./files/ssl/star.wmflabs.org.key [19:53:00] ./modules/secret/secrets/ssl/star.wmflabs.key [19:53:02] ./modules/secret/secrets/ssl/star.wmflabs.org.key [19:53:10] too many keys in the labs/private :D [19:53:42] at least they are the same [19:55:57] http://en.wikipedia.beta.wmflabs.org/wiki/Special:UserLogin is 503'ing for me. [19:56:12] matt_flaschen: have you looked at logstash for the exceptions ? :D [19:56:33] matt_flaschen: https://logstash-beta.wmflabs.org (it is a public, more or less looks like the one from prod) [19:56:44] there is even [19:56:50] the fatal monitor!!!!!!!!! https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/fatalmonitor [19:59:15] hashar, how do I figure out the URL responsible for the fatal? [20:00:06] seems to be anything hitting the database [20:00:28] hashar, no, I just posted something to the DB. [20:00:35] Catchable fatal error: Argument 1 passed to DatabaseBase::getCacheSetOptions() must implement interface IDatabase, undefined variable given in /srv/mediawiki/php-master/includes/db/Database.php on line 3834 [20:00:42] Aaron just made a patch for that [20:01:00] https://gerrit.wikimedia.org/r/#/c/246457/ ? ? [20:01:20] ahh [20:01:21] https://gerrit.wikimedia.org/r/#/c/247344/ [20:02:39] I am off to bed [20:02:57] +2d [20:03:02] beta should be fixed soon [20:03:11] Thanks, Reedy [20:05:24] hashar: Maybe we should issue a brand new cert for *.beta.wmflabs.org? [20:05:30] Rather than reuse this old thing? [20:10:54] ostriches: I gave up myself and advised Tyler to avoid touching the topic [20:10:59] maybe we can talk about it tomorrow ? [20:11:18] either wont fix / switch off the prototls stuff from beta [20:11:32] or go with some self signed cert [20:12:29] I say just make a new self signed cert. [20:13:35] ah here is one of the task https://phabricator.wikimedia.org/T50501 [20:13:45] that list a bunch of wild card certs we would need [20:13:55] brandon black has good ideas and potentially ambition to help "fix" the https issue on beta [20:14:06] the private one at https://phabricator.wikimedia.org/T97593 has an estimation of the cost ( not cheap) [20:14:13] this one: https://phabricator.wikimedia.org/T97593 [20:14:15] yeah [20:15:35] two+ years ago i proposed to use our own ca to generate all the wild card https://lists.wikimedia.org/pipermail/labs-l/2013-May/001212.html [20:16:26] but as i understood it if you trust the fake CA, that becomes a trusted CA for any domain [20:16:49] so if the CA is leaked somehow, you are in troubles [20:16:54] then i gave up [20:17:23] greg-g: I like bblack's ideas. [20:19:05] I also think we should remove > 50% of the people who have access to deployment-prep. [20:19:09] any opinions on if i can get a deploy window between the end of parsoid deploy (14:00PDT) and the begining of SWAT (16:00PDT)? basically we have limited time on a machine for testing elasticsearch replication to labs (the machine is to be given back next week). Due to no deploy last week we need to ship ~7 patches so we can test this week, rather than waiting for the thursday train(which would limit tests to only next week). [20:19:14] I see basically zero reason people need shell access. [20:19:41] *most people [20:19:43] tbh i can't think of the last time i needed deployment-prep shell access [20:20:02] i was going to set something up a few weeks ago (kafka) but then after poking realized someone else already did :) [20:20:13] ebernhardson: sure, add to calendar, plz [20:20:19] ebernhardson: removed :p [20:22:01] greg-g: thnaks, adding [20:23:07] greg-g: https://phabricator.wikimedia.org/T100837#1701062 ? [20:24:37] ostriches: I have removed a few more :-} [20:27:15] ostriches: yes, it probably just inherated from the ssl task [20:27:25] okie dokie [20:27:43] I think that shell access is needed to check the Beta databases. [20:27:51] And test things in eval.php. [20:27:51] 10Beta-Cluster-Infrastructure, 6Labs, 10Labs-Infrastructure, 6operations, 5WMF-NDA: On deployment-prep, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837#1736162 (10demon) [20:28:02] 10Beta-Cluster-Infrastructure, 6Labs, 10Labs-Infrastructure, 6operations, 5WMF-NDA: On deployment-prep, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837#1321787 (10demon) >>! In T100837#1701062, @Krenair wrote: > Can we make this task public? Done. [20:28:18] we should put back Special:Sql [20:28:23] matt_flaschen: Let me clarify...root access :) [20:28:41] https://www.mediawiki.org/wiki/Extension:Asksql [20:29:02] hashar: :P [20:29:18] ostriches: analytics folks probably needs root [20:32:31] nyway sleep [20:32:48] going to watch some K2 / Everest climbing videos [20:33:40] where is beta logstash? [20:33:46] * ostriches should know this, but can't find it [20:33:58] https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/default [20:34:00] Ah, logstash-beta [20:34:38] bd808: Would you believe me if I told you beta hasn't had an OOM in a week? [20:35:01] I couldn't find any this morning either [20:35:09] kinds weird [20:35:18] *kinda [20:36:52] hashar: Please can you merge https://gerrit.wikimedia.org/r/#/c/226634/ please. [20:37:06] hashar: and https://gerrit.wikimedia.org/r/#/c/247309/ please. [20:37:24] bd808: I found 2 oom's from syslog. [20:37:49] Heh, there's my 2 from experimenting. [20:38:53] I'm going to go with the "beta just hasn't been running out of memory" for now. There's no reason it shouldn't work...the other one clearly does. [20:51:19] ostriches: worst case scenario is that the initial counts in prod are wrong and we have to adjust the rules. seems very low risk [20:51:57] indeed [21:12:39] 10Beta-Cluster-Infrastructure, 6Labs, 10Labs-Infrastructure, 6operations: On deployment-prep, add warning text + labs Term of Uses link to the motd files - https://phabricator.wikimedia.org/T100837#1736320 (10Krenair) [21:14:17] Could someone merge https://gerrit.wikimedia.org/r/#/c/226634/ and https://gerrit.wikimedia.org/r/#/c/247309/ please. [21:20:48] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #39: 04FAILURE in 4 min 47 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/39/ [21:23:05] why does http://en.wikibooks.beta.wmflabs.org/wiki/Main_Page have Beta Commons logo? [21:23:52] Luke081515|away: ^ :) [22:07:52] ostriches: is this like...pending changesets get included kind of deal? https://gerrit.wikimedia.org/r/#/c/227489/ [22:17:58] PROBLEM - Puppet failure on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:19:01] chasemp: pretty sure yeah [22:19:57] I'm wondering if that cleans itself up [22:20:09] i.e. changes just pollute forever even after merge? [22:39:28] PROBLEM - Puppet failure on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:39:50] 6Release-Engineering-Team, 6Phabricator, 10Traffic, 6operations: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1736632 (10chasemp) >>! In T100519#1717447, @mmodell wrote: > @chasemp: Is there anything remaining for this to be completed? Feel free to claim and close this task.... [22:40:41] 10Continuous-Integration-Config: gate-and-submit for TimedMediaHandler runs phplint after running parallel-lint in composer test - https://phabricator.wikimedia.org/T114210#1736634 (10Paladox) 5Open>3Resolved a:3Paladox Closing as resolved since when this was merged https://gerrit.wikimedia.org/r/#/c/24729... [22:54:35] 6Release-Engineering-Team, 6operations: deployment: user trebuchet gets added and removed from group wikidev on every puppet run - https://phabricator.wikimedia.org/T115760#1736723 (10chasemp) [22:54:47] 6Release-Engineering-Team, 6operations: deployment: user trebuchet gets added and removed from group wikidev on every puppet run - https://phabricator.wikimedia.org/T115760#1732096 (10chasemp) @thcipriani Can we remove trebuchet user from wikidev all together? [22:57:37] legoktm@gallium:~$ ./test_extension.py [22:57:37] Use zuul-test-repo now. See https://gerrit.wikimedia.org/r/#/c/247031/ [22:57:38] woot [22:57:41] jzerebecki: ^ :) [22:59:12] 5Continuous-Integration-Scaling, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1736733 (10chasemp) We actually decided on this at the offsite :) This box can go in labs-support and there is no issue w/ current releng permissions translating. It wil... [23:03:47] 5Continuous-Integration-Scaling, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1736740 (10chasemp) a:5chasemp>3RobH >>! In T95046#1228381, @RobH wrote: > I'm going to assign this to chase, only while the discussion is pending about the networking... [23:15:47] 6Release-Engineering-Team, 6operations: deployment: user trebuchet gets added and removed from group wikidev on every puppet run - https://phabricator.wikimedia.org/T115760#1736808 (10chasemp) a:5chasemp>3thcipriani >>! In T115760#1736723, @chasemp wrote: > @thcipriani > > Can we remove trebuchet user f... [23:19:23] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Proposal: Force any WARNINGs on Beta Cluster to fail completely - https://phabricator.wikimedia.org/T115633#1736821 (10MaxSem) I had an idea to introduce a config setting for that into MW proper, to easily enable developers to see problems, no matter of... [23:38:24] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<50.00%)