[00:55:38] PROBLEM - Puppet staleness on integration-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [04:46:31] Could someone look into the patches I've submitted at https://phabricator.wikimedia.org/tag/mediawiki-codesniffer/ [04:56:06] Q: At what time branching for Tuesday happens? [04:56:28] TasneemLo: check with legoktm [04:57:17] hi [04:57:33] TasneemLo: sorry, they're on my list. I'll try and get to them tomorrow [04:57:44] legoktm, thanks :) [04:59:26] 6Release-Engineering-Team, 6Project-Creators, 15User-greg: Create #beta-cluster-reproducible tag and rename #beta-cluster to #beta-cluster-infrastructure - https://phabricator.wikimedia.org/T114486#1701273 (10Legoktm) +1 [05:07:56] 10MediaWiki-Codesniffer, 5Patch-For-Review: @codingStandardsIgnoreStart Generic.Files.LineLength.TooLong doesn't seem to work in tests/phpunit/includes/password/BcryptPasswordTest.php - https://phabricator.wikimedia.org/T114213#1701283 (10Legoktm) >>! In T114213#1699845, @Umherirrender wrote: > maybe this comm... [05:41:21] kart_: about 11am Pacific, ish [05:41:33] kart_: it's done by mukunda, manually kicked off [06:35:24] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [07:41:24] 6Release-Engineering-Team, 6Project-Creators, 15User-greg: Create #beta-cluster-reproducible tag and rename #beta-cluster to #beta-cluster-infrastructure - https://phabricator.wikimedia.org/T114486#1701396 (10hashar) @Luke081515 has an excellent idea! For greg-g plan antoine-approve [07:48:47] 10Deployment-Systems, 5Patch-For-Review: Adopt Semantic Versioning format for WMF deploy branches beginning with 1.27.0-wmf.1 - https://phabricator.wikimedia.org/T67306#1701401 (10hashar) We still have people consuming the WMF wiki API and we had a few scripts breaking due to the version scheme change. Maybe... [07:52:41] 10Browser-Tests, 6Release-Engineering-Team, 6Security: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1701402 (10hashar) Given 1.x to 1.6.x is trivial, you might as well bump the repositories directly to 1.6.x :-D [08:23:12] 10MediaWiki-Codesniffer, 5Patch-For-Review: @codingStandardsIgnoreStart only works with // comments - https://phabricator.wikimedia.org/T114213#1701423 (10Amire80) [08:29:00] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #742: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/742/ [09:05:48] 10MediaWiki-Codesniffer: Handle comments with multiple # - https://phabricator.wikimedia.org/T114633#1701497 (10TasneemLo) 3NEW a:3TasneemLo [09:08:24] 10Continuous-Integration-Infrastructure, 7HHVM, 7WorkType-Maintenance: Uninformative unit test failure - https://phabricator.wikimedia.org/T114522#1701515 (10hashar) Awesome @anomie [09:16:12] (03PS1) 10TasneemLo: Handle multiple # comments in Space Before Comment [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/243619 (https://phabricator.wikimedia.org/T114633) [09:16:37] (03CR) 10Hashar: [C: 04-1] "I don't think we are ready for composer yet. The idea is to use mediawiki/vendor for wmf branches and composer for master. Jan is doing" [integration/config] - 10https://gerrit.wikimedia.org/r/243597 (https://phabricator.wikimedia.org/T100654) (owner: 10Paladox) [09:20:24] (03CR) 10Hashar: [C: 04-1] [timeline] Update jenkins tests (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/243596 (owner: 10Paladox) [09:20:46] (03PS3) 10Hashar: [timeline] Update jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/243596 (owner: 10Paladox) [09:21:00] (03CR) 10Hashar: "Still depends on https://gerrit.wikimedia.org/r/#/c/243596/" [integration/config] - 10https://gerrit.wikimedia.org/r/243596 (owner: 10Paladox) [09:23:31] greg-g: thanks! [09:37:17] (03CR) 10Hashar: "Depends on" [integration/config] - 10https://gerrit.wikimedia.org/r/243394 (owner: 10Paladox) [09:37:24] (03CR) 10Hashar: [C: 04-1] [cldr] Update jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/243394 (owner: 10Paladox) [09:39:50] (03CR) 10Hashar: [C: 04-1] "Depends on https://gerrit.wikimedia.org/r/#/c/243208/" [integration/config] - 10https://gerrit.wikimedia.org/r/243209 (https://phabricator.wikimedia.org/T90943) (owner: 10Paladox) [09:40:42] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow: Flow browser test: hover failure in IE in flow_logged_in test - https://phabricator.wikimedia.org/T69723#1701597 (10zeljkofilipin) [09:40:57] (03PS3) 10Hashar: [TwnMainPage] Remove phpcs test, update tests [integration/config] - 10https://gerrit.wikimedia.org/r/243203 (https://phabricator.wikimedia.org/T90943) (owner: 10Paladox) [09:41:10] (03CR) 10Hashar: [C: 04-1] "Depends on https://gerrit.wikimedia.org/r/#/c/243159/" [integration/config] - 10https://gerrit.wikimedia.org/r/243203 (https://phabricator.wikimedia.org/T90943) (owner: 10Paladox) [09:41:34] (03PS4) 10Hashar: [TranslationNotifications] Update tests [integration/config] - 10https://gerrit.wikimedia.org/r/243209 (https://phabricator.wikimedia.org/T90943) (owner: 10Paladox) [09:43:06] (03CR) 10Hashar: [C: 032] "Good, lets get this in. We can refine with follow up patches as needed." [integration/jenkins] - 10https://gerrit.wikimedia.org/r/243092 (owner: 10John Vandenberg) [09:43:51] (03Merged) 10jenkins-bot: Fix Python 3 syntax and import errors [integration/jenkins] - 10https://gerrit.wikimedia.org/r/243092 (owner: 10John Vandenberg) [10:04:42] (03PS5) 10Hashar: Add composer-test for 4 extensions [integration/config] - 10https://gerrit.wikimedia.org/r/243156 (owner: 10Siebrand) [10:05:31] (03CR) 10Hashar: [C: 032] "Rebased. I have dropped configuration for source patches that are not ready yet:" [integration/config] - 10https://gerrit.wikimedia.org/r/243156 (owner: 10Siebrand) [10:06:15] (03PS6) 10Hashar: Add composer-test for 4 extensions [integration/config] - 10https://gerrit.wikimedia.org/r/243156 (owner: 10Siebrand) [10:06:24] (03CR) 10Hashar: [C: 032] Add composer-test for 4 extensions [integration/config] - 10https://gerrit.wikimedia.org/r/243156 (owner: 10Siebrand) [10:07:19] (03Merged) 10jenkins-bot: Add composer-test for 4 extensions [integration/config] - 10https://gerrit.wikimedia.org/r/243156 (owner: 10Siebrand) [10:25:21] (03Abandoned) 10Siebrand: Update tests for Translate [integration/config] - 10https://gerrit.wikimedia.org/r/243169 (owner: 10Siebrand) [10:48:48] (03PS2) 10Hashar: [ActivityMonitor] Add extension-unittests-generic test [integration/config] - 10https://gerrit.wikimedia.org/r/242178 (owner: 10Paladox) [10:49:01] (03CR) 10Hashar: [C: 032] [ActivityMonitor] Add extension-unittests-generic test [integration/config] - 10https://gerrit.wikimedia.org/r/242178 (owner: 10Paladox) [10:50:00] (03Merged) 10jenkins-bot: [ActivityMonitor] Add extension-unittests-generic test [integration/config] - 10https://gerrit.wikimedia.org/r/242178 (owner: 10Paladox) [11:11:14] 10Browser-Tests, 6Release-Engineering-Team, 6Security: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1701722 (10zeljkofilipin) @hashar: repositories that are using 0.x will be updated to the latest released version at the time of upd... [11:12:33] (03PS2) 10Hashar: operations/puppet now has strict puppet-lint [integration/config] - 10https://gerrit.wikimedia.org/r/243185 (https://phabricator.wikimedia.org/T87132) [11:19:01] (03CR) 10Hashar: "Unfortunately a couple issues have been introduced:" [integration/config] - 10https://gerrit.wikimedia.org/r/243185 (https://phabricator.wikimedia.org/T87132) (owner: 10Hashar) [11:37:07] (03CR) 10Hashar: Job for apps/android/java-mwapi (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/243181 (https://phabricator.wikimedia.org/T108735) (owner: 10Hashar) [12:30:18] (03PS2) 10Hashar: Deleted failing Flow browsertests Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/242887 (https://phabricator.wikimedia.org/T94153) (owner: 10Zfilipin) [12:30:36] (03CR) 10Hashar: [C: 032] "Already deleted https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sa" [integration/config] - 10https://gerrit.wikimedia.org/r/242887 (https://phabricator.wikimedia.org/T94153) (owner: 10Zfilipin) [12:32:41] (03CR) 10Hashar: [C: 04-1] "Probably. Looks like LightweightRDFa is no more active." [integration/config] - 10https://gerrit.wikimedia.org/r/239189 (https://phabricator.wikimedia.org/T93727) (owner: 10Paladox) [12:39:29] (03Merged) 10jenkins-bot: Deleted failing Flow browsertests Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/242887 (https://phabricator.wikimedia.org/T94153) (owner: 10Zfilipin) [12:42:06] 10MediaWiki-Codesniffer, 5Patch-For-Review: @codingStandardsIgnoreStart only works with // comments - https://phabricator.wikimedia.org/T114213#1701882 (10Amire80) [12:42:09] 10MediaWiki-Codesniffer, 5MW-1.27-release-notes, 5Patch-For-Review, 5WMF-deploy-2015-10-06_(1.27.0-wmf.2): @codingStandardsIgnoreStart doesn't seem to help avoid phpcs errors inside comments - https://phabricator.wikimedia.org/T113852#1701883 (10Amire80) [12:42:12] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 5MW-1.26-release, 5MW-1.27-release-notes, and 3 others: Fix Generic.Files.LineLength errors in MediaWiki core by breaking long lines - https://phabricator.wikimedia.org/T102614#1701881 (10Amire80) [12:42:59] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 5MW-1.26-release, 5MW-1.27-release-notes, and 3 others: Fix Generic.Files.LineLength errors in MediaWiki core by breaking long lines - https://phabricator.wikimedia.org/T102614#1678517 (10Amire80) Merging https://gerrit.wikimedia.org/r/#/c/241501/ wi... [12:44:12] 10MediaWiki-Codesniffer, 7Upstream: @codingStandardsIgnoreStart only works with // comments - https://phabricator.wikimedia.org/T114213#1701887 (10Amire80) [12:46:14] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata, 5Patch-For-Review: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1701891 (10hashar) Seems good now isn't it ? [12:51:16] 10Continuous-Integration-Infrastructure, 10Wikimedia-Fundraising-CiviCRM: Job wikimedia-fundraising-civicrm fails on branch civi-4.6.9-deploy causing force merges that breaks Zuul scheduler - https://phabricator.wikimedia.org/T114165#1701893 (10hashar) Happy you managed to fix it :-} [13:01:37] (03CR) 10JanZerebecki: [C: 04-1] "This repo is included in the Wikidata build, thus it has nothing to do with vendor.git. At least until we move from having the Wikidata bu" [integration/config] - 10https://gerrit.wikimedia.org/r/243597 (https://phabricator.wikimedia.org/T100654) (owner: 10Paladox) [13:16:02] 10Beta-Cluster, 10MediaWiki-extensions-BounceHandler: Beta config needs update (Undefined variable: wmgVERPsecret) - https://phabricator.wikimedia.org/T75757#1701924 (1001tonythomas) I see two occurrences of $wgVERPsecret in which the labs one is commented out. In production, it gets copied as $wmgVERPSecret.... [13:16:49] 10Continuous-Integration-Infrastructure, 10MediaWiki-RfCs, 5Patch-For-Review: [RFC] Optional Travis integration for Jenkins - https://phabricator.wikimedia.org/T114421#1701928 (10hashar) I have seen the npm-travis change and vetoed it on the basis that relying on Gerrit/Github replication to trigger build in... [13:35:56] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata, 5Patch-For-Review: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1701956 (10JanZerebecki) [13:36:24] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata, 5Patch-For-Review: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1653521 (10JanZerebecki) Not yet, I updated the description with what still needs to be done. [13:38:30] zeljkof: I am cherry picking the contint/ruby stuff on slaves [13:38:37] and will see what happens [13:38:45] hashar: great [13:38:53] I am still flipping papers :( [13:40:17] !log stopping puppetmaster on integration. Out of memory [13:40:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:41:48] PROBLEM - Puppet failure on integration-slave-precise-1011 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:43:02] PROBLEM - Puppet failure on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:45:32] !log lets funk it https://soundcloud.com/professorkliq/wire-flashing-lights?in=professorkliq/sets/wire-and-flashing-lights-ep [13:45:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:45:39] !sal [13:45:39] https://tools.wmflabs.org/sal/releng [13:45:47] qa-morebots: good bot [13:45:48] I am a logbot running on tools-exec-1216. [13:45:48] Messages are logged to https://tools.wmflabs.org/sal/releng. [13:45:48] To log a message, type !log . [13:51:31] zeljkof: they pass :-} [13:53:23] Wow! [13:53:33] hashar: the patch is ready? [13:54:01] yeah [13:54:04] I think it is good enough [13:54:14] I am too lazy to rethink about it [13:55:42] we need a Swift cluster [13:55:46] to send logs / artifacts too [13:55:47] grr [13:56:45] RECOVERY - Puppet failure on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [14:08:04] RECOVERY - Puppet failure on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [14:15:20] zeljkof: jaime merged them in ! [14:15:35] hashar: yeah! [14:23:34] (03PS1) 10Hashar: nodepool: add ruby packages to instances [integration/config] - 10https://gerrit.wikimedia.org/r/243655 [14:23:43] zeljkof: ^^ :D [14:24:37] (03CR) 10jenkins-bot: [V: 04-1] nodepool: add ruby packages to instances [integration/config] - 10https://gerrit.wikimedia.org/r/243655 (owner: 10Hashar) [14:24:48] hashar: wow, it looks so easy... ;) [14:25:51] yeah [14:25:56] well [14:25:56] note the -1 [14:26:10] * hashar whistles [14:26:28] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/243655 (owner: 10Hashar) [14:33:05] (03CR) 10Hashar: [C: 032] nodepool: add ruby packages to instances [integration/config] - 10https://gerrit.wikimedia.org/r/243655 (owner: 10Hashar) [14:34:09] (03Merged) 10jenkins-bot: nodepool: add ruby packages to instances [integration/config] - 10https://gerrit.wikimedia.org/r/243655 (owner: 10Hashar) [14:34:19] !log nodepool image-update wmflabs-eqiad ci-jessie-wikimedia [14:34:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:34:30] zeljkof: ^^ that is more or less all what is needed to refresh an image iirc [14:34:38] from https://wikitech.wikimedia.org/wiki/Nodepool#Manually_generate_a_new_snapshot [14:35:38] it is suppose to boot the image / pull integration/config and run puppet [14:36:49] (03PS3) 10Hashar: operations/puppet now has strict puppet-lint [integration/config] - 10https://gerrit.wikimedia.org/r/243185 (https://phabricator.wikimedia.org/T87132) [14:37:22] (03CR) 10Hashar: [C: 032] "https://gerrit.wikimedia.org/r/#/c/243653/ fix the last issue" [integration/config] - 10https://gerrit.wikimedia.org/r/243185 (https://phabricator.wikimedia.org/T87132) (owner: 10Hashar) [14:37:47] (03CR) 10jenkins-bot: [V: 04-1] operations/puppet now has strict puppet-lint [integration/config] - 10https://gerrit.wikimedia.org/r/243185 (https://phabricator.wikimedia.org/T87132) (owner: 10Hashar) [14:37:54] E: Unable to locate package ruby-bundler [14:37:57] I knew it [14:38:36] (03Merged) 10jenkins-bot: operations/puppet now has strict puppet-lint [integration/config] - 10https://gerrit.wikimedia.org/r/243185 (https://phabricator.wikimedia.org/T87132) (owner: 10Hashar) [14:43:32] (03PS1) 10Hashar: Remove pplint-HEAD from operations/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/243658 [14:43:43] (03CR) 10Hashar: [C: 032] Remove pplint-HEAD from operations/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/243658 (owner: 10Hashar) [14:45:36] !log rebooting integration-slave-jessie-1001.integration.eqiad.wmflabs , does not reply to ssh [14:45:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:46:39] * hashar blames ferm [14:46:57] (Not sure who does ldap/wmf access requests, except that I don't think it's Ops; please triage T114646 as appropriate.) [14:48:25] (03Merged) 10jenkins-bot: Remove pplint-HEAD from operations/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/243658 (owner: 10Hashar) [14:48:26] !log integration-slave-jessie-1001 fails ssh auth with: error: AuthorizedKeysCommand /usr/sbin/ssh-key-ldap-lookup returned status 1 [14:48:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:48:54] James_F: that is most certainly ops, and definitely not releng :-} [14:49:25] hashar: I was told the opposite in this channel just last week. [14:49:49] hashar: Historically it was done by R.eedy or o.striches [14:53:55] 6Release-Engineering-Team, 6Editing-Department: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1702072 (10Jdforrester-WMF) [14:54:10] 6Release-Engineering-Team, 6Editing-Department, 10Ops-Access-Requests, 6operations: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1702074 (10hashar) Seems a task for #ops-access-requests . I at least don't have any access to LDAP. James me... [14:54:18] James_F: I have added my random mumbling to it D [14:58:15] 10Deployment-Systems, 10Salt, 6operations, 5Patch-For-Review: [Trebuchet] Salt times out on parsoid restarts - https://phabricator.wikimedia.org/T63882#1702094 (10ArielGlenn) I've looked at this some over the weekend. Some notes: It looks like the timeout is not passed through from service-restart all th... [15:01:44] (03CR) 10Niedzielski: [C: 04-1] "thanks, 2 responses inline" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/243181 (https://phabricator.wikimedia.org/T108735) (owner: 10Hashar) [15:04:14] (03PS1) 10JanZerebecki: Add repo data-values/value-view [integration/config] - 10https://gerrit.wikimedia.org/r/243671 [15:08:32] (03CR) 10JanZerebecki: [C: 032] Add repo data-values/value-view [integration/config] - 10https://gerrit.wikimedia.org/r/243671 (owner: 10JanZerebecki) [15:09:13] (03CR) 10jenkins-bot: [V: 04-1] Add repo data-values/value-view [integration/config] - 10https://gerrit.wikimedia.org/r/243671 (owner: 10JanZerebecki) [15:15:57] hashar: any idea why the tox-jessie job is failing randomly on integration/config?: ERROR: could not install deps [15:16:17] (03CR) 10JanZerebecki: [V: 032] Add repo data-values/value-view [integration/config] - 10https://gerrit.wikimedia.org/r/243671 (owner: 10JanZerebecki) [15:16:42] (03CR) 10Hashar: Job for apps/android/java-mwapi (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/243181 (https://phabricator.wikimedia.org/T108735) (owner: 10Hashar) [15:17:03] (03CR) 10JanZerebecki: Add repo data-values/value-view [integration/config] - 10https://gerrit.wikimedia.org/r/243671 (owner: 10JanZerebecki) [15:17:08] (03CR) 10JanZerebecki: [C: 032] Add repo data-values/value-view [integration/config] - 10https://gerrit.wikimedia.org/r/243671 (owner: 10JanZerebecki) [15:18:03] (03Merged) 10jenkins-bot: Add repo data-values/value-view [integration/config] - 10https://gerrit.wikimedia.org/r/243671 (owner: 10JanZerebecki) [15:20:09] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: tox-jessie job is failing randomly on integration/config with could not install deps - https://phabricator.wikimedia.org/T114650#1702148 (10JanZerebecki) 3NEW [15:20:43] (03CR) 10Niedzielski: [C: 031] "lg2m! thank you for your patience" [integration/config] - 10https://gerrit.wikimedia.org/r/243181 (https://phabricator.wikimedia.org/T108735) (owner: 10Hashar) [15:21:51] !log reloading zuul for d944bbb..1272657 [15:21:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:24:36] Reedy: Any chance you could fix T114646 whilst people discuss who might theoretically be on-call to do it? :-) [15:25:13] James_F: We don't have the waiting period for this, do we? [15:25:37] (03CR) 10JanZerebecki: "Is puppetlint really a superset of puppet validate?" [integration/config] - 10https://gerrit.wikimedia.org/r/243658 (owner: 10Hashar) [15:26:19] hiring was the waiting period, I guess [15:26:35] Reedy: No. [15:26:50] heh [15:28:13] !log integration-slave-jessie-1001 pass puppet again. Caused by a conflict with the 'zip' package, solved by cherry picking https://gerrit.wikimedia.org/r/#/c/243674/ [15:28:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:28:47] !log integration-slave-jessie-1001 ssh reacheable [15:28:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:29:28] * Reedy greps where the ldap tools live now [15:30:50] 10Browser-Tests, 7Documentation, 7Easy, 7Software-Licensing: Ruby gem documentation should state license - https://phabricator.wikimedia.org/T94001#1702195 (10Revi) [15:31:01] ah, it is terbium, just not in path, and sudo ofc [15:31:54] off see you later [15:32:05] * greg-g waves to hashar [15:34:14] 6Release-Engineering-Team, 6Project-Creators, 15User-greg: Create #beta-cluster-reproducible tag and rename #beta-cluster to #beta-cluster-infrastructure - https://phabricator.wikimedia.org/T114486#1702212 (10greg) ok, enough +1s, I'll JFDI now. [15:37:36] (03PS4) 10Paladox: [timeline] Update jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/243596 [15:38:04] James_F: Don't suppose you remember seeing the actual instructions anywhere recently do you? [15:38:21] Reedy: No. :-( [15:38:38] 6Release-Engineering-Team, 6Editing-Department, 10Ops-Access-Requests, 6operations: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1702226 (10Krenair) There is an LDAP admins group (CC'd) on the servers (so ops should not have to handle thes... [15:38:40] There's the SVN page on wikitech, which is vaguely right, but that's not for adding someone to a group [15:38:52] Reedy: https://wikitech.wikimedia.org/wiki/LDAP is hilarious. [15:40:10] privileges: ['ALL = NOPASSWD: /usr/local/sbin/add-ldap-user', [15:40:10] 'ALL = NOPASSWD: /usr/local/sbin/delete-ldap-user', [15:40:10] 'ALL = NOPASSWD: /usr/local/sbin/modify-ldap-user', [15:40:10] 'ALL = NOPASSWD: /usr/local/bin/svn-group', [15:40:10] 'ALL = NOPASSWD: /usr/local/sbin/add-labs-user', [15:40:11] 'ALL = NOPASSWD: /usr/local/sbin/modify-ldap-group'] [15:40:32] modify-ldap-group then? [15:40:47] ah, yeah [15:40:50] I kind of wish we'd get rid of the wmf group and split it up into actual roles [15:40:50] there's the --add-members [15:41:34] Krenair: Even more process for no-one to follow? :-) [15:41:37] Reedy: mutante has it in his bash history [15:41:42] no more process [15:41:46] just not an organisation-based group [15:41:58] here he is :) [15:42:16] [terbium:~] $ ldaplist -l group wmf [15:42:18] I can do it [15:42:21] to check: [15:42:22] or... [15:42:32] mutante|away: yeah, we'd found which shell script it's under :P [15:42:46] [terbium:~] $ #sudo modify-ldap-group --addmembers [15:42:52] sudo /usr/local/sbin/modify-ldap-group --addmembers= [15:43:02] yes [15:44:03] reedy@terbium:~$ sudo /usr/local/sbin/modify-ldap-group --addmembers=Ferdbold,Tchanders,DLynch wmf [15:44:03] DLynch doesn't exist, and won't be added to the group. [15:44:07] James_F: 66% done [15:44:08] :P [15:44:14] 6Release-Engineering-Team, 6Editing-Department, 10Ops-Access-Requests, 6operations: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1702246 (10Jdforrester-WMF) >>! In T114646#1702226, @Krenair wrote: > There is an LDAP admins group (CC'd) on... [15:44:23] 6Release-Engineering-Team, 6Editing-Department, 10LDAP-Access-Requests: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1702248 (10Jdforrester-WMF) [15:44:30] Reedy: Erm. [15:44:37] Reedy: He exists on gerrit… [15:44:50] Reedy: it's probably the "WMF" thing [15:44:52] and a space [15:45:03] reedy@terbium:~$ ldaplist -l passwd DLynch [15:45:03] reedy@terbium:~$ [15:45:13] Ho-hum. [15:45:21] 6Release-Engineering-Team, 6Project-Creators, 15User-greg: Create #beta-cluster-reproducible tag and rename #beta-cluster to #beta-cluster-infrastructure - https://phabricator.wikimedia.org/T114486#1702251 (10greg) Everything done except moving those two tasks into -reproducible (it's created) because @Jdfor... [15:45:34] 6Release-Engineering-Team, 6Project-Creators, 15User-greg: Create #beta-cluster-reproducible tag and rename #beta-cluster to #beta-cluster-infrastructure - https://phabricator.wikimedia.org/T114486#1702253 (10greg) 5Open>3Resolved [15:46:26] Reedy: i didn't even know you can add multiple at once with commas .. [15:46:29] Reedy: Oh, he's not pushed a patch yet; do you think that would auto-create him or something? [15:46:32] all good? [15:46:48] mutante: it doesn't seem to have done anything :P [15:47:23] oh, it has [15:47:25] [terbium:~] $ ldaplist -l group wmf | grep [15:47:28] because fuck alphasort [15:47:36] yea, isnt that weird [15:47:43] they are all alpha until recently [15:47:52] like 5 users ago it started to just append instead [15:47:56] :D [15:48:09] (03PS1) 10Paladox: [Echo] Add composer-test test [integration/config] - 10https://gerrit.wikimedia.org/r/243679 [15:48:43] 6Release-Engineering-Team, 6Editing-Department, 10LDAP-Access-Requests: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1702257 (10Reedy) Ferdbold and Tchanders are added. DLynch apparently doesn't exist. According to ldap at least [15:49:33] James_F: do the other 2 have shell accounts? [15:50:03] Reedy: No? [15:50:04] Reedy: Oh, he's not pushed a patch yet; do you think that would auto-create him or something? [15:50:04] no [15:50:09] Hmm. [15:50:15] LDAP user existence comes from wikitech signups [15:50:19] (or SVN import) [15:50:27] Is it possible that his wikitech sign-up didn't work? [15:50:43] gerrit just uses LDAP to auth (same way as various other misc services), it doesn't create users [15:51:12] So what creates the user? Extension:LDAP on wikitech? [15:51:30] https://wikitech.wikimedia.org/wiki/Special:UserRights/DLynch shows he exists [15:51:46] Yeah. [15:52:17] Reedy: Sorry, this was meant to be a five-second thing. [15:52:22] heh [15:52:32] Did he enter a different shell username? [15:52:36] into wikitech [15:52:50] James_F, Extension:LdapAuthentication [15:53:06] Reedy: Ah, maybe. Try "kemayo"? [15:53:29] yup, that's it [15:53:34] Yep [15:53:34] Ha. [15:53:37] uid: kemayo [15:53:41] mail: dlynch@wikimedia.org [15:53:41] cn: DLynch [15:53:58] reedy@terbium:~$ sudo /usr/local/sbin/modify-ldap-group --addmembers=kemayo wmf [15:53:58] reedy@terbium:~$ [15:54:00] No data is good data [15:54:10] Woo. [15:54:38] 6Release-Engineering-Team, 6Editing-Department, 10LDAP-Access-Requests: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1702301 (10Reedy) [15:54:54] 6Release-Engineering-Team, 6Editing-Department, 10LDAP-Access-Requests: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1702302 (10Reedy) 5Open>3Resolved a:3Reedy kemayo added too [15:55:46] Reedy, James_F: In future when this comes up, you can just `ldaplist -l passwd | grep -C 20 -i ` [15:55:52] to go through all users [15:55:58] lol [15:56:03] in this case my was 'dlynch' [15:56:09] Krenair: Neat. [15:56:16] Not really. [15:56:25] Reedy: Thank you. [15:56:27] :P [15:59:50] heh "not really" [16:02:10] 6Release-Engineering-Team, 6Editing-Department, 10LDAP-Access-Requests: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1702326 (10Jdforrester-WMF) Thank you! [16:03:14] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:03:14] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:03:14] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:05:02] (03CR) 10Niedzielski: "apps-android-java-mwapi[0] and apps-android-wikipedia-test[1] appear to be working correctly. poking for a +2..." [integration/config] - 10https://gerrit.wikimedia.org/r/243181 (https://phabricator.wikimedia.org/T108735) (owner: 10Hashar) [16:05:09] ^^^ hello! https://gerrit.wikimedia.org/r/#/c/243181 has been tested and is ready for +2 [16:10:20] (03PS1) 10Paladox: [Translate] Add composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/243683 [16:14:59] 10Continuous-Integration-Infrastructure, 10MediaWiki-RfCs, 5Patch-For-Review: [RFC] Optional Travis integration for Jenkins - https://phabricator.wikimedia.org/T114421#1702398 (10greg) @cscott: You only describe what you see as things that'd be "fixed" with Travis integration while not taking a critical look... [16:26:15] James_F: btw, if you still have thoughts on the #beta-cluster-reproducible tag name, feel free to comment on that closed task or here or whatever [16:30:19] 10Continuous-Integration-Config, 5Patch-For-Review: Whitelist Paladox in Zuul configuration - https://phabricator.wikimedia.org/T114528#1702507 (10Paladox) [16:31:05] 10Continuous-Integration-Config, 5Patch-For-Review: Whitelist Paladox in Zuul configuration - https://phabricator.wikimedia.org/T114528#1698487 (10Paladox) Was trying to add a subscriber that I thought wasent in the list. [16:32:11] 10Continuous-Integration-Config, 10Gerrit: Provide access to "check zend" - https://phabricator.wikimedia.org/T114632#1702523 (10greg) [16:35:11] greg-g: We can always rename it later. I'm not that fussed. :-) [16:36:14] kk [16:42:15] 6Release-Engineering-Team, 6Project-Creators, 15User-greg: Create #beta-cluster-reproducible tag and rename #beta-cluster to #beta-cluster-infrastructure - https://phabricator.wikimedia.org/T114486#1702583 (10greg) (ok, I moved 'em, we can rename the tag later if needed) [16:44:19] 10Continuous-Integration-Config: Provide access to "check zend" - https://phabricator.wikimedia.org/T114632#1702600 (10greg) [16:44:59] 10Continuous-Integration-Config: Provide access to "check zend" for @MarcoAurelio - https://phabricator.wikimedia.org/T114632#1702605 (10greg) [16:48:47] Krenair: what do you think of removing obsolete/inactive repos from github (eg the ones listed here: https://phabricator.wikimedia.org/T102920 ) ? [16:48:54] cc others [16:48:56] :) [16:49:40] as long as they are primarily stored in gerrit and the github repo is just a mirror, sure [16:51:12] that's what I mean, yeah [17:58:36] 6Release-Engineering-Team, 6Editing-Department, 10LDAP-Access-Requests: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1702990 (10hashar) Sorry @Jdforrester-WMF for the tongue-in-cheek earlier in the IRC channel. I am happy to know such reque... [17:59:37] 6Release-Engineering-Team, 6Project-Creators, 15User-greg: Create #beta-cluster-reproducible tag and rename #beta-cluster to #beta-cluster-infrastructure - https://phabricator.wikimedia.org/T114486#1703000 (10hashar) Sounds good, @greg would you mind announcing it? Perhaps on QA list, the private engineering... [18:00:47] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-09-29: [Task] Set up Jenkins to run unit tests against a combined repo+client setup. - https://phabricator.wikimedia.org/T112136#1703020 (10JanZerebecki) I decided is was best to first change ever job using repo only to r... [18:01:24] 10Continuous-Integration-Config: Whitelist @MarcoAurelio in Zuul configuration so he can 'recheck' / 'check zend' - https://phabricator.wikimedia.org/T114632#1703035 (10hashar) [18:02:29] 10Continuous-Integration-Config, 10Wikidata: [Task] Wikidata: Set up Jenkins to run unit tests also against repo without client - https://phabricator.wikimedia.org/T114690#1703043 (10JanZerebecki) 3NEW [18:02:42] 6Release-Engineering-Team, 6Project-Creators, 15User-greg: Create #beta-cluster-reproducible tag and rename #beta-cluster to #beta-cluster-infrastructure - https://phabricator.wikimedia.org/T114486#1703049 (10greg) [18:02:48] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-09-29: [Task] Set up Jenkins to run unit tests against a combined repo+client setup. - https://phabricator.wikimedia.org/T112136#1703050 (10JanZerebecki) a:3JanZerebecki [18:03:34] 10Continuous-Integration-Config, 10Wikidata: [Task] Wikidata: Set up Jenkins to run unit tests also against repo without client - https://phabricator.wikimedia.org/T114690#1703043 (10JanZerebecki) [18:03:37] 6Release-Engineering-Team, 6Project-Creators, 15User-greg: Create #beta-cluster-reproducible tag and rename #beta-cluster to #beta-cluster-infrastructure - https://phabricator.wikimedia.org/T114486#1697304 (10greg) [18:06:53] 10Continuous-Integration-Infrastructure: run extension jobs with and without optional dependencies - https://phabricator.wikimedia.org/T114691#1703063 (10JanZerebecki) 3NEW [18:07:57] 6Release-Engineering-Team, 6Project-Creators, 15User-greg: Create #beta-cluster-reproducible tag and rename #beta-cluster to #beta-cluster-infrastructure - https://phabricator.wikimedia.org/T114486#1703071 (10greg) >>! In T114486#1703000, @hashar wrote: > Sounds good, @greg would you mind announcing it? Perh... [18:08:40] 10Continuous-Integration-Config, 10Wikidata: [Task] Wikibase: Set up Jenkins to run unit tests also against repo without client - https://phabricator.wikimedia.org/T114690#1703073 (10JanZerebecki) [18:33:50] 10Deployment-Systems, 3Scap3, 6Discovery: Create deployment for wikimedia/portals - https://phabricator.wikimedia.org/T114694#1703156 (10mmodell) I'd really like to see this use #scap3 for deployment. [18:37:01] 10Deployment-Systems, 3Scap3, 6Discovery: Create deployment for wikimedia/portals - https://phabricator.wikimedia.org/T114694#1703171 (10mmodell) a:3mmodell [18:37:26] 7Blocked-on-RelEng, 10Deployment-Systems, 3Scap3, 6Discovery: Create deployment for wikimedia/portals - https://phabricator.wikimedia.org/T114694#1703137 (10mmodell) [18:38:18] 10Deployment-Systems, 3Scap3: Migrate the MW weekly train deploy to scap3 - https://phabricator.wikimedia.org/T114313#1703179 (10Smalyshev) [18:38:20] 7Blocked-on-RelEng, 10Deployment-Systems, 3Scap3, 6Discovery: Create deployment for wikimedia/portals - https://phabricator.wikimedia.org/T114694#1703178 (10Smalyshev) [18:39:04] 6Release-Engineering-Team, 6Editing-Department, 10LDAP-Access-Requests: Please add Frédéric Bolduc, Thalia Chan, and David Lynch to ldap/wmf - https://phabricator.wikimedia.org/T114646#1703198 (10Aklapper) >>! In T114646#1702246, @Jdforrester-WMF wrote: > After a brief discussion with @greg, there's now #LDA... [19:02:36] 10Continuous-Integration-Infrastructure, 10MediaWiki-RfCs, 5Patch-For-Review: [RFC] Optional Travis integration for Jenkins - https://phabricator.wikimedia.org/T114421#1703273 (10cscott) It is indeed "pretty easy to solve" -- I just run `npm run travis` locally before I C+2 any patch. And other projects jus... [19:09:17] 10Continuous-Integration-Infrastructure, 10MediaWiki-RfCs, 5Patch-For-Review: [RFC] Optional Travis integration for Jenkins - https://phabricator.wikimedia.org/T114421#1703276 (10greg) >>! In T114421#1703273, @cscott wrote: > It is indeed "pretty easy to solve" -- I just run `npm run travis` locally before I... [19:11:01] 10Continuous-Integration-Config: Whitelist @MarcoAurelio in Zuul configuration so he can 'recheck' / 'check zend' - https://phabricator.wikimedia.org/T114632#1703282 (10hashar) Hola Marco, apparently you have been whitelisted on Sep. 17th with the change https://gerrit.wikimedia.org/r/#/c/239146/1/zuul/layout.ya... [19:11:26] 10Continuous-Integration-Infrastructure, 10MediaWiki-RfCs, 5Patch-For-Review: [RFC] Optional Travis integration for Jenkins - https://phabricator.wikimedia.org/T114421#1703284 (10greg) So, summary: For this RFC to go forward, as any RFC, there should be a breakdown of costs/benefits by the proposer. [19:11:29] 10Browser-Tests, 10Continuous-Integration-Config, 5Patch-For-Review, 7Regression: mwext-mw-selenium is slowing down gate-and-submit pipeline - https://phabricator.wikimedia.org/T113501#1703285 (10dduvall) It's worth pointing out that we're still seeing sharp peaks in max and median gating and there doesn't... [19:18:42] 10Continuous-Integration-Infrastructure, 10MediaWiki-RfCs, 5Patch-For-Review: [RFC] Optional Travis integration for Jenkins - https://phabricator.wikimedia.org/T114421#1703298 (10cscott) FWIW, a "travis bot" like the existing `jenkins-bot` could also be run as an unsupported service on labs. My assumption i... [19:40:34] 10Continuous-Integration-Infrastructure, 10MediaWiki-RfCs, 5Patch-For-Review: [RFC] Optional Travis integration for Jenkins - https://phabricator.wikimedia.org/T114421#1703355 (10cscott) >>! In T114421#1703284, @greg wrote: > So, summary: For this RFC to go forward, as any RFC, there should be a breakdown of... [20:15:30] !sal [20:15:30] https://tools.wmflabs.org/sal/releng [20:15:39] I should change that alias [20:15:47] so it includes "thank you bd808 for the new sal" [20:16:14] "experimental" sal ;) [20:16:26] yeah it is dead right now [20:16:42] the on-wiki one right? not mine? [20:16:49] that one https://tools.wmflabs.org/sal/ [20:17:07] but #wikimedia-labs was talking about some issue with lighttlpd on tools labs [20:17:11] yikes. I'll see if I can make it restart [20:18:20] hmm, yeah. my bash tool is down too [20:18:22] !log nodepool image-update wmflabs-eqiad ci-jessie-wikimedia [20:18:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:18:59] bd808: seems it is being deal with [20:19:12] bd808: can the tool suck down irc logs that it missed and fill in missed data? [20:19:22] ...automagically? ;) [20:25:50] !log Image ci-jessie-wikimedia-1444076296 in wmflabs-eqiad is ready. It provides bundler/ruby [20:25:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:26:10] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 7Tracking: tracking: Cleanup contint puppet manifests so they are easier to reuse / split slaves by functions - https://phabricator.wikimedia.org/T110864#1703507 (10hashar) [20:31:34] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: Factor out ruby stuff from contint puppet manifests - https://phabricator.wikimedia.org/T110865#1703548 (10hashar) Checked, the CI Nodepool instance comes with ruby2.1. Not sure of the impact it has on our gems [20:33:15] (03PS1) 10Hashar: Add tox-jessie to pywikibot/core [integration/config] - 10https://gerrit.wikimedia.org/r/243798 [20:34:33] (03CR) 10Hashar: [C: 032] Add tox-jessie to pywikibot/core [integration/config] - 10https://gerrit.wikimedia.org/r/243798 (owner: 10Hashar) [20:35:36] (03Merged) 10jenkins-bot: Add tox-jessie to pywikibot/core [integration/config] - 10https://gerrit.wikimedia.org/r/243798 (owner: 10Hashar) [21:52:59] 6RelEng-Admin, 15User-greg: Prepare Q1 - 2015/16 Quarterly Report - Due Monday Oct 5th - https://phabricator.wikimedia.org/T113115#1703760 (10greg) 5Open>3Resolved [21:58:50] !log deployment-prep Fixed l10nupdate - I broke it with https://gerrit.wikimedia.org/r/#/c/228126/ - created https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep/host/deployment-bastion [21:58:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:00:18] !log deployment-prep Doing a manual run of l10nupdate [22:00:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:08:45] 10Continuous-Integration-Infrastructure, 10Parsoid, 10RESTBase, 6Services, 7Parsoid-Tests: Move Parsoid and RESTBase testing from Travis CI to our Jenkins - https://phabricator.wikimedia.org/T78410#1703790 (10GWicke) As an update, this is the current status: We are testing each commit in restbase and var... [22:10:52] !log deployment-prep Renamed deployment-bastion:/srv/l10nupdate/mediawiki/extensions/BlueSpiceExtensions to OldBlueSpiceExtensions, was not in a clean state for l10nupdate [22:10:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:12:47] looks like it's actually rebuilding caches now [22:13:16] Actually, I'm wondering when that BlueSpiceExtensions thing happened [22:13:37] If it was long enough ago and l10nupdate is supposed to be the thing dealing with magic words, it might be the cause of https://phabricator.wikimedia.org/T106125 [22:14:55] 10th April [22:15:34] But in that ticket I found that there were issues going back to October last year [22:16:25] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<55.56%) [22:17:41] huh, deployment-bastion:/var is full [22:17:58] PROBLEM - Puppet failure on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:19:31] /var/log/account is taking up half a gig [22:19:32] pacct [22:19:47] yeah. I nuke them when the disk is full and I notice [22:19:50] Feel like I've seen that one before [22:20:15] its a long standing problem on the old images with small /var partitions [22:20:25] !log deployment-prep deleted deployment-bastion:/var/log/account files to free up some space on /var [22:20:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:20:36] oh dear [22:20:43] l10nupdate failed due to that disk space usage [22:21:06] are you running the "real" l10nupdate job? [22:21:11] yes [22:21:12] it had got all the way to rebuilding hsb [22:21:28] it does all of its work in /var/something [22:21:46] there once was symlink that moved that over to the /mnt partition [22:22:25] since beta cluster runs master of everything there really shouldn't be any work for the l10nupdate script to do [22:22:39] since it pulls in master and overlays that on the deployed branches [22:22:58] I think we did have it setup just for testing at some point though [22:26:27] Interesting that l10nupdate does not appear to perform a rebuild for en-rtl. [22:26:35] I wonder if that's the root of T106125. [22:39:28] PROBLEM - Puppet failure on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:43:49] bah, got to "pap" before hitting the disk space limit this time [22:47:38] patched it to try /srv instead of /var/lib [22:47:52] (live hack on deployment-puppetmaster, will clean up after I'm done) [22:53:24] does anyone know why the normal SAL doesn't work in beta? [22:53:35] (or maybe it's just logging to somewhere I'm not expected?) [22:53:38] expecting* [22:56:04] Krenair: https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:56:19] (also: https://tools.wmflabs.org/sal/releng ) [22:56:37] ok, nothing from the servers though? [22:56:48] oh, like from scap? I guess not [22:57:01] when I sync stuff in production, unless I explicitly set it to suppress, it logs publicly [22:57:09] Krenair: there's https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep#Server_admin_log , but it's probably different [22:57:30] Krenair: right, not that [22:57:44] Krenair: also https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Log_Files , sorry if I'm misdirecting 8-) [22:57:54] not sure why/why not (other than scap being too noisy being every 10 minutes) [22:58:05] yeah those aren't what I'm looking for, thanks anyway spagewmf [22:58:11] ah, good point [22:58:22] we could set the automatic ones to not log though [22:59:15] right [22:59:24] create a task :) [22:59:45] (I think I like this idea, after we think through all the various gotchas, if there are many) [23:06:18] (03PS1) 10Legoktm: Revert "Disable hhvm tests for wikimedia/slimapp" [integration/config] - 10https://gerrit.wikimedia.org/r/243828 (https://phabricator.wikimedia.org/T107677) [23:11:27] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [23:30:19] https://phabricator.wikimedia.org/P2156 [23:32:15] keyholder shiz? [23:32:27] cc thcipriani ^ [23:32:28] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Phase out yamllint jobs (tracking) - https://phabricator.wikimedia.org/T95890#1704100 (10jayvdb) [23:32:30] 10Continuous-Integration-Infrastructure, 10Wikimedia-Fundraising, 5Patch-For-Review: Add YAML lint test in wikimedia/fundraising/tools to remove the Jenkins job 'yamllint' - https://phabricator.wikimedia.org/T114470#1704097 (10jayvdb) 5Open>3Resolved a:3hashar [23:32:46] greg-g, I don't think l10nupdate uses keyholder? [23:32:59] it must, to sync around the data [23:33:30] * thcipriani looks [23:34:48] I ran the sync-dir manually anyway [23:37:47] Project beta-scap-eqiad build #72975: 04FAILURE in 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/72975/ [23:41:35] huh, the l10nupdate key in the home directory definitely doesn't match the one in /etc/ssh/userkeys/l10nupdate [23:41:56] not a keyholder thing, but a weird thing. [23:42:53] Yippee, build fixed! [23:42:54] Project beta-scap-eqiad build #72976: 09FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/72976/ [23:52:44] 10Continuous-Integration-Infrastructure, 10MediaWiki-General-or-Unknown: mediawiki-config needs better coverage - https://phabricator.wikimedia.org/T114725#1704146 (10MaxSem) 3NEW