[00:41:36] 06Release-Engineering-Team, 06Operations, 10vm-requests, 07Security-General: New ganeti VM for MW release pipeline work - https://phabricator.wikimedia.org/T163743#3208457 (10RobH) While it needs gerrit for basic cloning/fetch, it won't care if it is within its same datacenter, correct? If it is fetching... [00:43:51] 06Release-Engineering-Team, 06Operations, 10vm-requests, 07Security-General: New ganeti VM for MW release pipeline work - https://phabricator.wikimedia.org/T163743#3208461 (10demon) >>! In T163743#3208457, @RobH wrote: > While it needs gerrit for basic cloning/fetch, it won't care if it is within its same... [01:55:28] Hello releng people [01:55:46] Did something change in Zuul/Jenkins/CI that affects the $wgDBname that phpunit tests are run with? [03:20:59] (03PS1) 10Aude: Bump Wikidata to wmf/1.29.0-wmf.21 [tools/release] - 10https://gerrit.wikimedia.org/r/350112 [03:31:15] (03CR) 10Aude: [C: 032] Bump Wikidata to wmf/1.29.0-wmf.21 [tools/release] - 10https://gerrit.wikimedia.org/r/350112 (owner: 10Aude) [03:31:49] (03Merged) 10jenkins-bot: Bump Wikidata to wmf/1.29.0-wmf.21 [tools/release] - 10https://gerrit.wikimedia.org/r/350112 (owner: 10Aude) [06:46:58] !log uncherry-pick f6ce64e99a and 225b8d4e82 (T161563) [06:47:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:47:02] T161563: Use gzip for logstash - https://phabricator.wikimedia.org/T161563 [07:55:08] 06Release-Engineering-Team, 06Operations, 10vm-requests, 07Security-General: New ganeti VM for MW release pipeline work - https://phabricator.wikimedia.org/T163743#3207976 (10fgiunchedi) >>! In T163743#3208393, @demon wrote: >>>! In T163743#3208387, @RobH wrote: >> I'd suggest we use the hostname jenkins-j... [08:05:22] 10Deployment-Systems, 10Scap, 13Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3208930 (10fgiunchedi) @thcipriani package built and updated in reprepro [08:07:06] 06Release-Engineering-Team, 06Operations, 10vm-requests, 07Security-General: New ganeti VM for MW release pipeline work - https://phabricator.wikimedia.org/T163743#3208931 (10hashar) We have CI hosts like contint1001 / contint2001. What about a generic name like: `contint1002.eqiad.wmnet` ? For network t... [08:21:16] (03PS3) 10Hashar: [UploadLogo] switch to npm job [integration/config] - 10https://gerrit.wikimedia.org/r/349910 [08:21:18] (03PS4) 10Hashar: [ScanSet] switch to npm job [integration/config] - 10https://gerrit.wikimedia.org/r/349914 [08:21:20] (03PS3) 10Hashar: [RandomFeaturedUser] switch to npm job [integration/config] - 10https://gerrit.wikimedia.org/r/349916 [08:21:22] (03PS3) 10Hashar: [OfflineImportLexicon] switch to npm job [integration/config] - 10https://gerrit.wikimedia.org/r/349918 [08:23:43] (03PS2) 10Hashar: [UserStatus] switch to npm job [integration/config] - 10https://gerrit.wikimedia.org/r/349905 [08:23:45] (03PS2) 10Hashar: [WikiTwidget] switch to npm job [integration/config] - 10https://gerrit.wikimedia.org/r/349902 [08:23:47] (03PS2) 10Hashar: [PictureGames] switch to npm job [integration/config] - 10https://gerrit.wikimedia.org/r/349899 [08:37:43] (03PS1) 10Hashar: [SkelJS] drop jshint/jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/350128 [08:40:23] 10Continuous-Integration-Config, 13Patch-For-Review: Remove jshint from V+1 as well as V+2 pipes for repos that use eslint now - https://phabricator.wikimedia.org/T154924#3208989 (10hashar) 05Open>03Resolved a:03hashar The Jenkins job that runs jshint has been removed from all the repositories. They now... [08:40:59] (03CR) 10Hashar: [C: 032] [SkelJS] drop jshint/jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/350128 (owner: 10Hashar) [08:41:56] (03Merged) 10jenkins-bot: [SkelJS] drop jshint/jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/350128 (owner: 10Hashar) [08:43:14] (03PS1) 10Hashar: Remove mwgate-jshint [integration/config] - 10https://gerrit.wikimedia.org/r/350129 [08:43:26] 10Continuous-Integration-Config, 05MW-1.27-release (WMF-deploy-2016-03-08_(1.27.0-wmf.16)), 13Patch-For-Review: Convert all repos to use npm Jenkins job with jsonlint and eslint - https://phabricator.wikimedia.org/T119973#3208996 (10hashar) The Jenkins job that runs jshint has been removed from all the repos... [09:11:08] (03PS1) 10Hashar: build: jsonlint/jshint via npm [integration/docroot] - 10https://gerrit.wikimedia.org/r/350134 [09:11:27] (03PS1) 10Hashar: integration/docroot: jshint/jsonlint -> npm [integration/config] - 10https://gerrit.wikimedia.org/r/350135 [09:11:32] (03CR) 10jerkins-bot: [V: 04-1] build: jsonlint/jshint via npm [integration/docroot] - 10https://gerrit.wikimedia.org/r/350134 (owner: 10Hashar) [09:12:34] (03CR) 10Hashar: [C: 032] integration/docroot: jshint/jsonlint -> npm [integration/config] - 10https://gerrit.wikimedia.org/r/350135 (owner: 10Hashar) [09:12:37] (03CR) 10Hashar: [C: 032] Remove mwgate-jshint [integration/config] - 10https://gerrit.wikimedia.org/r/350129 (owner: 10Hashar) [09:13:25] (03PS1) 10Hashar: Archive integration/raita [integration/config] - 10https://gerrit.wikimedia.org/r/350137 [09:13:41] (03CR) 10Hashar: [C: 032] Archive integration/raita [integration/config] - 10https://gerrit.wikimedia.org/r/350137 (owner: 10Hashar) [09:14:34] (03Merged) 10jenkins-bot: Remove mwgate-jshint [integration/config] - 10https://gerrit.wikimedia.org/r/350129 (owner: 10Hashar) [09:14:36] (03Merged) 10jenkins-bot: integration/docroot: jshint/jsonlint -> npm [integration/config] - 10https://gerrit.wikimedia.org/r/350135 (owner: 10Hashar) [09:14:53] (03Merged) 10jenkins-bot: Archive integration/raita [integration/config] - 10https://gerrit.wikimedia.org/r/350137 (owner: 10Hashar) [09:15:36] (03CR) 10Hashar: "recheck" [integration/docroot] - 10https://gerrit.wikimedia.org/r/350134 (owner: 10Hashar) [09:17:53] (03CR) 10Hashar: [C: 032] "That is solely to get rid of the Jenkins job "jshint". We can look at migrating to eslint later on." [integration/docroot] - 10https://gerrit.wikimedia.org/r/350134 (owner: 10Hashar) [09:19:16] (03PS1) 10Hashar: UserGroups: drop jshint from the check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/350139 [09:21:00] (03Merged) 10jenkins-bot: build: jsonlint/jshint via npm [integration/docroot] - 10https://gerrit.wikimedia.org/r/350134 (owner: 10Hashar) [09:21:06] (03CR) 10jenkins-bot: build: jsonlint/jshint via npm [integration/docroot] - 10https://gerrit.wikimedia.org/r/350134 (owner: 10Hashar) [09:33:33] (03PS1) 10Hashar: wikimania-scholarship: switch to npm [integration/config] - 10https://gerrit.wikimedia.org/r/350142 [09:34:11] (03PS1) 10Hashar: Remove wikimedia/iegreview (moved to Phabricator) [integration/config] - 10https://gerrit.wikimedia.org/r/350144 [09:35:48] (03CR) 10Hashar: [C: 032] UserGroups: drop jshint from the check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/350139 (owner: 10Hashar) [09:36:09] (03CR) 10Hashar: [C: 032] wikimania-scholarship: switch to npm [integration/config] - 10https://gerrit.wikimedia.org/r/350142 (owner: 10Hashar) [09:37:10] (03Merged) 10jenkins-bot: UserGroups: drop jshint from the check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/350139 (owner: 10Hashar) [09:37:30] (03Merged) 10jenkins-bot: wikimania-scholarship: switch to npm [integration/config] - 10https://gerrit.wikimedia.org/r/350142 (owner: 10Hashar) [09:37:45] (03CR) 10Hashar: [C: 032] Remove wikimedia/iegreview (moved to Phabricator) [integration/config] - 10https://gerrit.wikimedia.org/r/350144 (owner: 10Hashar) [09:38:52] (03Merged) 10jenkins-bot: Remove wikimedia/iegreview (moved to Phabricator) [integration/config] - 10https://gerrit.wikimedia.org/r/350144 (owner: 10Hashar) [09:40:55] (03PS1) 10Hashar: wikimedia/fundraising/tools: drop jshint/jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/350146 (https://phabricator.wikimedia.org/T119973) [09:41:10] (03CR) 10Hashar: [C: 032] wikimedia/fundraising/tools: drop jshint/jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/350146 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [09:42:07] (03Merged) 10jenkins-bot: wikimedia/fundraising/tools: drop jshint/jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/350146 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [09:45:13] (03PS1) 10Hashar: Remove all jshint references [integration/config] - 10https://gerrit.wikimedia.org/r/350147 (https://phabricator.wikimedia.org/T119973) [09:45:25] (03PS1) 10Hashar: Remove jshint entirely [integration/jenkins] - 10https://gerrit.wikimedia.org/r/350148 (https://phabricator.wikimedia.org/T119973) [09:47:07] (03CR) 10Hashar: [C: 032] Remove all jshint references [integration/config] - 10https://gerrit.wikimedia.org/r/350147 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [09:47:08] (03CR) 10Hashar: [C: 032] Remove jshint entirely [integration/jenkins] - 10https://gerrit.wikimedia.org/r/350148 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [09:47:57] (03Merged) 10jenkins-bot: Remove jshint entirely [integration/jenkins] - 10https://gerrit.wikimedia.org/r/350148 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [09:48:41] (03Merged) 10jenkins-bot: Remove all jshint references [integration/config] - 10https://gerrit.wikimedia.org/r/350147 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [09:52:49] (03PS1) 10Hashar: Remove jsonlint from check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/350152 (https://phabricator.wikimedia.org/T119973) [09:54:58] Hey. I don't know the process around beta cluster changes but is it okay if we merge this right now and just sync it in naos? https://gerrit.wikimedia.org/r/349984 [09:55:04] or I should wait until SWAT? [09:56:11] Amir1: never ever merge a change without deploying it [09:56:27] if someone runs scap or whatever, the change will end up being deployed [09:57:34] I know but I mean this patch is only for beta (but still needs deployment for sync-ness reasons) so still should be done in SWAT? [09:57:43] ah [09:57:47] or since it's not affecting prod we can do it right now [09:57:49] well if that is beta only, yeah I gess that can land [09:58:07] nice [09:58:07] though Kunal had some concerns about the .svg [09:58:15] but I guess that can be polished later [09:58:36] hmm, we compressed it it's only 200 bytes [09:58:39] :D [09:59:29] Amir1: well [09:59:57] shouldn't it be served by resource loader anyway? :) [10:03:32] (03PS1) 10Hashar: Remove 'jsonlint' job [integration/config] - 10https://gerrit.wikimedia.org/r/350154 (https://phabricator.wikimedia.org/T119973) [10:04:03] (03CR) 10Hashar: [C: 032] Remove jsonlint from check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/350152 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [10:05:25] (03Merged) 10jenkins-bot: Remove jsonlint from check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/350152 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [10:10:33] (03CR) 10Hashar: [C: 032] Remove 'jsonlint' job [integration/config] - 10https://gerrit.wikimedia.org/r/350154 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [10:10:49] (03PS1) 10ArielGlenn: enable tox for new repo operations/dumps/import-tools [integration/config] - 10https://gerrit.wikimedia.org/r/350156 [10:11:59] (03Merged) 10jenkins-bot: Remove 'jsonlint' job [integration/config] - 10https://gerrit.wikimedia.org/r/350154 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [10:12:13] (03PS2) 10Hashar: enable tox for new repo operations/dumps/import-tools [integration/config] - 10https://gerrit.wikimedia.org/r/350156 (owner: 10ArielGlenn) [10:12:20] (03CR) 10Hashar: [C: 032] "\O/" [integration/config] - 10https://gerrit.wikimedia.org/r/350156 (owner: 10ArielGlenn) [10:13:33] (03Merged) 10jenkins-bot: enable tox for new repo operations/dumps/import-tools [integration/config] - 10https://gerrit.wikimedia.org/r/350156 (owner: 10ArielGlenn) [10:19:56] Yippee, build fixed! [10:19:57] Project language-screenshots-VisualEditor » chrome,Windows 10,ci-jessie-wikimedia build #55: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/language-screenshots-VisualEditor/BROWSER=chrome,PLATFORM=Windows%2010,label=ci-jessie-wikimedia/55/ [10:28:22] (03PS1) 10Hashar: Unconfigure PdfBook, moved to Github [integration/config] - 10https://gerrit.wikimedia.org/r/350159 [10:28:24] (03PS1) 10Hashar: Drop mwgate-jsonlint from some repositories [integration/config] - 10https://gerrit.wikimedia.org/r/350160 (https://phabricator.wikimedia.org/T119973) [10:34:18] (03PS2) 10Hashar: Drop mwgate-jsonlint from some repositories [integration/config] - 10https://gerrit.wikimedia.org/r/350160 (https://phabricator.wikimedia.org/T119973) [10:35:15] (03PS1) 10Hashar: Unconfigure WYSIWYG (archived) [integration/config] - 10https://gerrit.wikimedia.org/r/350161 [10:35:23] (03CR) 10Hashar: [C: 032] Unconfigure PdfBook, moved to Github [integration/config] - 10https://gerrit.wikimedia.org/r/350159 (owner: 10Hashar) [10:35:41] (03CR) 10Hashar: [C: 032] Unconfigure WYSIWYG (archived) [integration/config] - 10https://gerrit.wikimedia.org/r/350161 (owner: 10Hashar) [10:36:10] (03CR) 10Hashar: [C: 032] Drop mwgate-jsonlint from some repositories [integration/config] - 10https://gerrit.wikimedia.org/r/350160 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [10:36:37] (03Merged) 10jenkins-bot: Unconfigure PdfBook, moved to Github [integration/config] - 10https://gerrit.wikimedia.org/r/350159 (owner: 10Hashar) [10:37:09] (03Merged) 10jenkins-bot: Drop mwgate-jsonlint from some repositories [integration/config] - 10https://gerrit.wikimedia.org/r/350160 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [10:37:31] (03Merged) 10jenkins-bot: Unconfigure WYSIWYG (archived) [integration/config] - 10https://gerrit.wikimedia.org/r/350161 (owner: 10Hashar) [10:39:04] (03PS1) 10Hashar: Remove Zuul template mwgate-jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/350162 [10:40:54] (03CR) 10Hashar: [C: 032] Remove Zuul template mwgate-jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/350162 (owner: 10Hashar) [10:41:55] (03Merged) 10jenkins-bot: Remove Zuul template mwgate-jsonlint [integration/config] - 10https://gerrit.wikimedia.org/r/350162 (owner: 10Hashar) [10:51:23] (03PS1) 10Hashar: Remove mwgate-jsonlint from mw-checks-test [integration/config] - 10https://gerrit.wikimedia.org/r/350166 (https://phabricator.wikimedia.org/T119973) [10:54:07] (03PS2) 10Hashar: Remove mwgate-jsonlint from mw-checks-test [integration/config] - 10https://gerrit.wikimedia.org/r/350166 (https://phabricator.wikimedia.org/T119973) [10:55:46] Yippee, build fixed! [10:55:47] Project language-screenshots-VisualEditor » chrome,Windows 10,ci-jessie-wikimedia build #56: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/language-screenshots-VisualEditor/BROWSER=chrome,PLATFORM=Windows%2010,label=ci-jessie-wikimedia/56/ [11:01:48] (03CR) 10Hashar: [C: 032] Remove mwgate-jsonlint from mw-checks-test [integration/config] - 10https://gerrit.wikimedia.org/r/350166 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [11:02:54] (03Merged) 10jenkins-bot: Remove mwgate-jsonlint from mw-checks-test [integration/config] - 10https://gerrit.wikimedia.org/r/350166 (https://phabricator.wikimedia.org/T119973) (owner: 10Hashar) [11:03:27] 10Continuous-Integration-Config, 05MW-1.27-release (WMF-deploy-2016-03-08_(1.27.0-wmf.16)), 13Patch-For-Review: Convert all repos to use npm Jenkins job with jsonlint and eslint - https://phabricator.wikimedia.org/T119973#3209294 (10hashar) Last job to migrated is wikimedia-fundraising-crm-jsonlint though th... [11:15:29] hashar: it seems not, at least for echo notification icons :/ [11:16:12] I don't know why they decided that way (I know there were some troubles for defining new bundles of icons inside OOjs UI but I can't say that was the case) [12:48:36] (03PS1) 10Hashar: Zend 5.5 composer jobs for civicrm [integration/config] - 10https://gerrit.wikimedia.org/r/350192 (https://phabricator.wikimedia.org/T163781) [12:50:33] (03CR) 10Hashar: [C: 032] "Job deployed and experimental. will let us test the few patches I have sent against wikimedia/fundraising/crm." [integration/config] - 10https://gerrit.wikimedia.org/r/350192 (https://phabricator.wikimedia.org/T163781) (owner: 10Hashar) [12:51:47] (03Merged) 10jenkins-bot: Zend 5.5 composer jobs for civicrm [integration/config] - 10https://gerrit.wikimedia.org/r/350192 (https://phabricator.wikimedia.org/T163781) (owner: 10Hashar) [13:34:46] 10Beta-Cluster-Infrastructure: Request for importer on zhwp beta - https://phabricator.wikimedia.org/T163784#3209825 (10vjudge404) [14:29:35] PROBLEM - Host deployment-phab02 is DOWN: CRITICAL - Host Unreachable (10.68.19.232) [15:04:49] If I want to test a config change by deploying to beta cluster, is there docs available on how to do that? [15:06:26] Can I just make a commit to the labs config files in operations/mediawiki-config.git and then things automatically happen? [15:07:17] bawolff: yes, but policy asks to deploy it, to have a prod/repo parity [15:07:32] RainbowSprinkles looks like upstream have begun finishing the elasticsearch implementation :), https://gerrit-review.googlesource.com/#/c/104516/ which will be in gerrit 2.14.1 anyways :) [15:07:38] bawolff: a Jenkins job will automatically pick it and deploy to labs [15:07:59] technically its a secret-ish thing, should I maybe put it into the equivalent of /private on deployment-tin.deployment-prep.eqiad.wmflabs ? [15:08:02] 10Continuous-Integration-Config, 10Gerrit, 10MediaWiki-extensions-AccountAudit, 06Project-Admins: Archive the AccountAudit extension - https://phabricator.wikimedia.org/T163791#3210080 (10Jdforrester-WMF) [15:08:24] or at least it will be secret in production [15:08:34] It was deployed to production but exploded logstash [15:17:18] I guess to do that I would need to be added to the deployment-prep project? [15:20:15] meh, I'll just put it in the super-public config file [15:22:49] 10Continuous-Integration-Config, 10Gerrit, 10MediaWiki-extensions-AccountAudit, 06Project-Admins: Archive the AccountAudit extension - https://phabricator.wikimedia.org/T163791#3210153 (10Aklapper) https://www.mediawiki.org/wiki/Gerrit/Inactive_projects has a list of things to do [15:22:52] 06Release-Engineering-Team, 06Operations, 10vm-requests, 07Security-General: New ganeti VM for MW release pipeline work - https://phabricator.wikimedia.org/T163743#3210152 (10demon) >>! In T163743#3208931, @hashar wrote: > We have CI hosts like contint1001 / contint2001. What about a generic name like: `c... [15:23:37] 06Release-Engineering-Team, 06Operations, 10vm-requests, 07Security-General: New ganeti VM for MW release pipeline work - https://phabricator.wikimedia.org/T163743#3210166 (10demon) I'd be fine with something like mwreleases1001! [15:28:40] Dereckson: Does the deploy labs changes for parody policy also apply to changes where the intention is a quick test and then revert probably ten minutes later? [15:30:35] What I'd like is the possibility to livetest something on labs cluster without merging in prod [15:31:00] apparently, that also would be convenient for you [15:31:05] We could make a labs branch! [15:31:14] JK THAT'S A TERRIBLE IDEA REMEMBER WHEN WE DID THAT FOR PUPPET? [15:31:16] :p [15:31:35] Dereckson: yeah, appearently its badform to live test on production [15:31:55] bawolff: Production is the best place to test ;-) [15:32:30] hashar did you notice the mysql upgrade taking long with ci? [15:32:52] I got an update on the jenkins slave and it freezes on setting up mysql 5.55 [15:32:54] So, to be concrete, I'd like to live test https://gerrit.wikimedia.org/r/#/c/350231/ on labs for like 5 minutes [15:32:55] its finished now. [15:33:14] Do I need to actually deploy that, or is there a way to live test? [15:33:56] Additionally, it seems like people are deploying all sorts of mysql changes, and I'd like to avoid annoying them if possible [15:34:22] although there is some urgency to this as its to address a high priority ongoing issue on frwiki [15:40:49] 06Release-Engineering-Team, 06Operations, 10vm-requests, 07Security-General: New ganeti VM for MW release pipeline work - https://phabricator.wikimedia.org/T163743#3210254 (10RobH) >>! In T163743#3208931, @hashar wrote: > We have CI hosts like contint1001 / contint2001. What about a generic name like: `co... [16:06:41] Hey releng people, I'm going to disable instance creation/deletion for a bit (10 minutes ish) as soon as puppet swat finishes. https://phabricator.wikimedia.org/T163402 [16:06:52] This is to prevent a longer outage tomorrow when some switch maintenance happens. [16:07:06] Sorry for the short notice. This may result in people kvetching about CI [16:15:29] weee [16:15:39] so, you know, a day ending in y [16:42:23] PROBLEM - Puppet errors on saucelabs-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:42:31] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:42:51] PROBLEM - Puppet errors on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:43:46] PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:44:18] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:44:48] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:44:56] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:45:49] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:46:00] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:46:55] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:46:59] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Reading-Infrastructure-Team-Backlog, and 3 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2857076 (10NHarateh_WMF) Was in "Next up" on Trending-Service workboard... [16:47:16] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Reading-Infrastructure-Team-Backlog, and 3 others: Setup trending service CI - https://phabricator.wikimedia.org/T151469#2818282 (10NHarateh_WMF) Was in "Next up" on Trending-Service workboard as of 04/... [16:47:28] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:47:40] PROBLEM - Puppet errors on deployment-salt02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:47:46] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:48:04] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:48:08] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:48:12] PROBLEM - Puppet errors on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:48:20] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:48:29] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:49:19] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:49:51] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:50:25] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:50:35] PROBLEM - Puppet errors on jenkinstest is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:50:40] PROBLEM - Puppet errors on deployment-zookeeper01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:50:42] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:51:00] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:17:26] RECOVERY - Puppet errors on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:17:27] RECOVERY - Puppet errors on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:19:48] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [17:19:49] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [17:19:56] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:19:56] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:21:55] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [17:21:55] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:31] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:31] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:48] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:48] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:52] RECOVERY - Puppet errors on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:52] RECOVERY - Puppet errors on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:03] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:03] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:32] RECOVERY - Puppet errors on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:32] RECOVERY - Puppet errors on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:46] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:46] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:18] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:18] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:18] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:18] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:27] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:27] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:41] RECOVERY - Puppet errors on deployment-zookeeper01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:41] RECOVERY - Puppet errors on deployment-zookeeper01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:41] RECOVERY - Puppet errors on jenkinstest is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:41] RECOVERY - Puppet errors on jenkinstest is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:49] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:49] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:58] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:58] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:10] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:10] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:29] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:29] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:33] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:33] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:43] RECOVERY - Puppet errors on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:43] RECOVERY - Puppet errors on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:51] RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:51] RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:07] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:08] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:12] RECOVERY - Puppet errors on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:12] RECOVERY - Puppet errors on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:18] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:18] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:50] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:50] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:40] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:40] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:58] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:59] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:44] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:19:14] 10Scap (Scap3-MediaWiki-MVP), 10scap2: Scap should touch symlinks when originals are touched - https://phabricator.wikimedia.org/T126306#3210921 (10demon) p:05Low>03High [18:19:27] thcipriani: I raised priority on that one ^ [18:19:30] People got bit again [18:19:43] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:25:01] Granted, we could *fix* the privatesettings instance if we just included the original file instead of a symlink [18:37:45] ok, I'm going to cause another nova outage. Will be done in time for the train. [19:02:07] ejegg: hello! I got you some patch for fundraising/crm to add a json linting script https://gerrit.wikimedia.org/r/#/q/topic:jsonlint+project:wikimedia/fundraising/crm :] [19:02:39] thanks! I'll take a look [19:02:42] ejegg: the intent is to get rid of the jenkins job that does it, and also add a parallel PHP linter ! [19:05:53] hashar: hmm, I think npm test does json linting for us in the DonationInterface repo [19:06:25] yeah, looks like grunt test includes that [19:06:31] ejegg: yup. But wikimedia/fundraising/crm has no npm / package.json / gruntfile.js etc [19:06:43] rightright... [19:07:20] it is not too hard to add all the js boiler plate though. Then I felt it was easier to keep a single test command: composer test [19:07:24] would it make sense to add that rather than a special php scrupt? [19:07:31] yup can do [19:07:52] this way I guess that will be consistent accross repos! [19:08:00] hmm, one single test command does sound nice though [19:08:26] then for CI it does not change much thing [19:08:41] we can well have json lint in the npm test job, at worth developers forget to run it locally [19:08:44] but CI will catch it [19:09:15] hashar: iirc CI instances don't rely on dns is that right? [19:09:54] ejegg: probably easier to use grunt-jsonlint like the other repos. Will craft a patch for that and you can decide which path you want :] [19:10:16] hashar: if it's no trouble, yeah, let's use grunt-jsonlint please! [19:10:26] hashar: thcipriani how is CI doing? [19:10:34] chasemp: they are added to jenkins with the 10.x IP , not the hostname fqdn [19:11:03] chasemp: if the underlying question is : should we stop populating DNS entries in designate for the contintcloud project: yeah that is probably worth a shot [19:11:14] ejegg: sold! [19:11:36] hashar: no, designate is broken atm adn I'm trying to make sure it's not effecting CI [19:12:47] there is no patches floating around for now, so hard to tell [19:13:34] hashar: mind putting something up to test? [19:19:45] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:20:17] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [19:20:57] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:21:58] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:22:56] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:28] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:34] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:46] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:52] RECOVERY - Puppet errors on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:29] RECOVERY - Puppet errors on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:15] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:50] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:26] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:38] RECOVERY - Puppet errors on deployment-zookeeper01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:40] RECOVERY - Puppet errors on jenkinstest is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:44] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:50] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:10] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:35] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:17] RECOVERY - Puppet errors on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:33] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:39] RECOVERY - Puppet errors on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:53] RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:29:09] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [19:29:10] RECOVERY - Puppet errors on castor is OK: OK: Less than 1.00% above the threshold [0.0] [19:29:12] RECOVERY - Puppet errors on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [19:29:20] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:32:00] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [19:32:20] RECOVERY - Puppet errors on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:42:44] PROBLEM - Puppet errors on integration-publishing is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:45:22] Project beta-scap-eqiad build #152546: 04FAILURE in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/152546/ [19:46:39] (03PS1) 10Hashar: npm job for wikimedia/fundraising/crm [integration/config] - 10https://gerrit.wikimedia.org/r/350272 (https://phabricator.wikimedia.org/T163781) [19:48:10] (03CR) 10Hashar: [C: 032] npm job for wikimedia/fundraising/crm [integration/config] - 10https://gerrit.wikimedia.org/r/350272 (https://phabricator.wikimedia.org/T163781) (owner: 10Hashar) [19:48:25] chasemp: sorry I went out for dinner [19:48:35] (out == 5 meters away actually) [19:49:13] Project beta-scap-eqiad build #152547: 04STILL FAILING in 1 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/152547/ [19:49:29] (03Merged) 10jenkins-bot: npm job for wikimedia/fundraising/crm [integration/config] - 10https://gerrit.wikimedia.org/r/350272 (https://phabricator.wikimedia.org/T163781) (owner: 10Hashar) [19:52:38] ejegg|semi: I sent another patch that uses npm / grunt-jsonlint. Will follow up next week I guess [19:52:44] ejegg|semi: vacations time for me [19:52:56] chasemp: I am off, Tyler should be able to assist [19:53:00] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:54:02] Project beta-scap-eqiad build #152548: 04STILL FAILING in 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/152548/ [19:54:24] PROBLEM - Puppet errors on deployment-stream is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [0.0] [19:55:47] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:05:39] Yippee, build fixed! [20:05:39] Project beta-scap-eqiad build #152549: 09FIXED in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/152549/ [20:09:56] ugh: https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm-jessie/11645/console [20:11:33] "Error: Wikibase requires MediaWiki 1.26 or above." [20:13:04] aude: any idea about that? ^ [20:13:43] it seems to be using the right version of wikibase ( 60264b1 ) [20:21:51] 10Scap (Scap3-MediaWiki-MVP), 10scap2: Scap should touch symlinks when originals are touched - https://phabricator.wikimedia.org/T126306#3211487 (10mmodell) @Jdlrobson this is probably what was causing us trouble during SWAT yesterday. [20:26:49] anyone know why wgVersion would be set incorrectly in CI ? [20:51:48] nvm found the problem: https://gerrit.wikimedia.org/r/#/c/350293/ [20:51:49] doh! [20:53:26] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T161733#3211674 (10mmodell) [21:06:43] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.30.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T162954#3211759 (10greg) [21:07:13] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3211761 (10greg) [21:23:06] greg-g: Ah, wmf.21 will be the last, no wmf.22? [21:23:37] well thats a bug milestone. [21:23:39] 1.30 [21:24:15] bug = big [21:25:28] 'Cos I made https://phabricator.wikimedia.org/project/profile/2730/ to unbreak RTB, which I suppose I now have to delete and remake under a new "MW-1.30-release". [21:31:18] James_F: sorry :/ [21:31:26] I wasn't thinking when I made the blocker tasks [21:31:31] paladox: Meh, no bigger than 1.20 (which wasn't :p) [21:31:40] greg-g: No worries at all. Fixed now. [21:31:40] oh [21:31:48] was 1.20 released in 2010? [21:31:53] greg-g: https://phabricator.wikimedia.org/project/view/2732/ [21:32:26] 2012 [21:32:43] https://www.mediawiki.org/wiki/Version_lifecycle [21:32:52] Not long ago then :). Now if someone upgraded from 1.20 to 1.30 that will be a big upgrade. [21:34:02] Nah. [21:34:09] Upgrades are easy, unless you're running 1.5.x! [21:34:18] 1.5.x -> 1.6.x was the only "hard" upgrade ever [21:34:34] oh [21:34:46] RainbowSprinkles: Wikia found issues with 1.19 -> 1.21. ;-) [21:34:49] by big i meant the ui, and other features :) [21:35:05] Also why is wikipedia founder launching another wiki news site when theres wikinews? [21:35:05] James_F: I'm going to keep my mouth shut [21:35:32] paladox: off-topic for here [21:35:36] RainbowSprinkles: No comment. :-D [21:35:47] ok [21:41:58] (03PS1) 10Arlolra: Remove the lint job from Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/350310 [21:43:40] (03CR) 10jerkins-bot: [V: 04-1] Remove the lint job from Parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/350310 (owner: 10Arlolra) [21:57:28] !log deployed ores cc12103 [21:57:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:59:08] Project selenium-Core » firefox,beta,Linux,BrowserTests build #392: 04FAILURE in 7 min 7 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/392/ [22:02:44] RainbowSprinkles: twentyafterfour: Published two patches to phab-ext for review (low priority) [22:03:12] Krinkle: cool [22:08:38] Krinkle: accepted, will deploy tomorrow [22:09:54] twentyafterfour: Thanks! [22:35:59] Should the "MW-1.30.0-wmf.1 deployment blockers" projects be children of "MW-1.30-release" (they're blockers of releases, it's the blocker of the overall release).