[00:10:13] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<100.00%) [00:12:46] !log legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf * [00:12:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:25:08] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [02:15:48] PROBLEM - Host deployment-redis02 is DOWN: CRITICAL - Host Unreachable (10.68.16.231) [02:15:50] PROBLEM - Host deployment-redis01 is DOWN: CRITICAL - Host Unreachable (10.68.16.177) [02:18:01] PROBLEM - Host deployment-dumps-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.21.153) [02:24:21] PROBLEM - Host deployment-puppetmaster02 is DOWN: CRITICAL - Host Unreachable (10.68.21.200) [02:36:48] Project beta-scap-eqiad build #211281: 04FAILURE in 3 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/211281/ [02:47:56] Yippee, build fixed! [02:47:56] Project beta-scap-eqiad build #211282: 09FIXED in 4 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/211282/ [05:46:24] 10Phabricator, 10Developer-Relations (Apr-Jun-2018), 10Patch-For-Review: Try to identify new developers (via assignee field) in Phab tasks and potentially follow up - https://phabricator.wikimedia.org/T195780#4236793 (10Dzahn) +---------------------------------------------------+ | userName... [06:15:40] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Pywikibot-core, 10Pywikibot-tests: Magul's quick tests doesn't run anymore - https://phabricator.wikimedia.org/T186208#4270931 (10Legoktm) OK the one thing I missed...what is the command to run the slow/full test suite? [07:52:00] zeljkof: good morning. The selenium tests being flappy for wikibase and others seems to have been tracked down by anomie :] [07:52:24] definitely due to a background job that invalidate a new session due to an old session being used [08:08:28] 10Gerrit: PUT "/r/accounts/self/status" - https://phabricator.wikimedia.org/T196869#4271040 (10hashar) [08:08:47] hashar: cool! [08:17:40] hashar: could you please https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/+/439466/ and https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/DidYouMean/+/439300/ ? [08:20:04] 10Gerrit, 10Release-Engineering-Team (Kanban): PUT "/r/accounts/self/status" - https://phabricator.wikimedia.org/T196869#4271058 (10hashar) [[ https://bugs.chromium.org/p/gerrit/issues/detail?id=3157 | Gerrit issue #3157 - Error in POST /accounts/self/preferences]] hints at permissions issue in All-Projects.gi... [08:21:14] 10Gerrit, 10Release-Engineering-Team (Kanban): PUT "/r/accounts/self/status" - https://phabricator.wikimedia.org/T196869#4271066 (10hashar) So I think it is all about changing rights of `root:root` files under `/srv/gerrit/git/All-Users.git` to `gerrit2:gerrit2` [08:22:53] (03CR) 10Hashar: [C: 032] Archive EditPageTracking extension [integration/config] - 10https://gerrit.wikimedia.org/r/439377 (https://phabricator.wikimedia.org/T190894) (owner: 10SamanthaNguyen) [08:24:25] (03Merged) 10jenkins-bot: Archive EditPageTracking extension [integration/config] - 10https://gerrit.wikimedia.org/r/439377 (https://phabricator.wikimedia.org/T190894) (owner: 10SamanthaNguyen) [08:32:11] 10Gerrit, 10Release-Engineering-Team (Kanban): PUT "/r/accounts/self/status" - https://phabricator.wikimedia.org/T196869#4271040 (10Legoktm) Same as {T196862} maybe? [08:34:46] 10Gerrit, 10Release-Engineering-Team (Kanban): PUT "/r/accounts/self/status" - https://phabricator.wikimedia.org/T196869#4271100 (10hashar) [08:34:48] 10Gerrit, 10Release-Engineering-Team: Some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196862#4271102 (10hashar) [08:35:19] 10Gerrit, 10Release-Engineering-Team (Kanban): PUT "/r/accounts/self/status" - https://phabricator.wikimedia.org/T196869#4271040 (10hashar) p:05Triage>03Unbreak! Yup. I have merged T196862 in there since I already have all the stacktraces. Thanks Paladox :] [08:35:56] 10Gerrit, 10Release-Engineering-Team (Kanban): In Gerrit some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196869#4271106 (10hashar) [09:07:23] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:14:02] (03PS1) 10Sebastian Berlin (WMSE): Whitelist Igor Leturia Azkarate [integration/config] - 10https://gerrit.wikimedia.org/r/439547 [09:46:19] (03PS1) 10Thiemo Kreuz (WMDE): Caused by switching the form to OOUI. [selenium] - 10https://gerrit.wikimedia.org/r/439559 [09:47:10] (03CR) 10jerkins-bot: [V: 04-1] Caused by switching the form to OOUI. [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [09:49:42] (03CR) 10Filippo Giunchedi: [C: 031] Mark repository as read only [software/tessera] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439469 (https://phabricator.wikimedia.org/T186096) (owner: 10MarcoAurelio) [09:49:57] (03PS2) 10Thiemo Kreuz (WMDE): Caused by switching the form to OOUI. [selenium] - 10https://gerrit.wikimedia.org/r/439559 [09:51:53] (03CR) 10jerkins-bot: [V: 04-1] Caused by switching the form to OOUI. [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [10:13:14] (03CR) 10Filippo Giunchedi: [C: 032] Mark repository as read only [software/tessera] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439469 (https://phabricator.wikimedia.org/T186096) (owner: 10MarcoAurelio) [10:13:20] (03CR) 10Filippo Giunchedi: [V: 032 C: 032] Mark repository as read only [software/tessera] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439469 (https://phabricator.wikimedia.org/T186096) (owner: 10MarcoAurelio) [10:35:03] (03PS3) 10Thiemo Kreuz (WMDE): Caused by switching the form to OOUI. [selenium] - 10https://gerrit.wikimedia.org/r/439559 [10:36:34] (03CR) 10jerkins-bot: [V: 04-1] Caused by switching the form to OOUI. [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [10:37:12] (03CR) 10Thiemo Kreuz (WMDE): "The same issue (CSS selectors for buttons change when a form starts using OOUI) happened in the Wikibase codebase a few times. I updated t" [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [10:56:45] 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username - https://phabricator.wikimedia.org/T138672#4271563 (10Paladox) Gerrit says it cannot find your user when doing owner:SMcCandlish but what is strange is your account should be created if it d... [10:57:47] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: In Gerrit some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196869#4271568 (10hashar) We need #operations to fix up permissions on cobalt.wikimedia.org Files under `/srv/gerrit/git/All-Users.git/` bei... [10:58:43] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: In Gerrit some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196869#4271583 (10Paladox) sudo chrown gerrit2:gerrit2 /srv/gerrit/git [11:01:23] (03CR) 10Hashar: [C: 032] .gitignore: Add several files to ignore [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/438233 (owner: 10Rafidaslam) [11:01:58] (03Merged) 10jenkins-bot: .gitignore: Add several files to ignore [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/438233 (owner: 10Rafidaslam) [11:02:39] 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username - https://phabricator.wikimedia.org/T138672#2406941 (10Dzahn) Hi all. I saw this ticket and checked Gerrit error logs for "candlish" and i found this which hopefully helps us to track it down... [11:03:37] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#4271590 (10Dzahn) [11:06:33] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4271595 (10Marostegui) [11:07:03] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#4271597 (10Paladox) I've asked upstream here https://groups.google.com/forum/#!topic/repo-discus... [11:07:38] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4271598 (10Paladox) This https://phabricator.wikimedia.org/D1067 will fix it so no more new notedb refs are cloned. [11:08:52] (03PS1) 10MarcoAurelio: Archive the ClientSide extension [integration/config] - 10https://gerrit.wikimedia.org/r/439572 (https://phabricator.wikimedia.org/T196617) [11:09:10] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4271605 (10Marostegui) >>! In T196840#4271598, @Paladox wrote: > This https://phabricator.wikimedia.org/D1067 will fix it so no more new notedb refs are cloned. When are you p... [11:09:28] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: In Gerrit some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196869#4271040 (10Dzahn) >>! In T196869#4271568, @hashar wrote: > We need #operations to fix up permissions on cobalt.wikimedia.org > > File... [11:09:53] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4271614 (10Paladox) Need someone to approve it, merge it and then i think @mmodell would have to deploy it. [11:10:26] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4271615 (10Marostegui) Excellent - thanks! :) [11:10:30] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: In Gerrit some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196869#4271616 (10Paladox) @dzahn though the objects inside that folder could be owned by root. [11:10:40] (03PS2) 10MarcoAurelio: Archive the ClientSide extension [integration/config] - 10https://gerrit.wikimedia.org/r/439572 (https://phabricator.wikimedia.org/T196617) [11:16:44] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: In Gerrit some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196869#4271646 (10Dzahn) >>! In T196869#4271568, @hashar wrote: > We need #operations to fix up permissions on cobalt.wikimedia.org Fixed.... [11:18:39] fixed the file permissions on Gerrit All-Users.git [11:19:02] which broke saving Gerrit user preferences for some users and should be fixed now [11:19:16] (03CR) 10WMDE-Fisch: "Zfilipin will fix the issue with the CI tomorrow. Let's wait for that then." [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [11:19:31] :) [11:19:46] legoktm you should be able to save your pref now ^^ :) [11:25:36] (03CR) 10Hashar: __init__.py: Supress unintentional 'true' print in `ansi_codes()` (032 comments) [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/438234 (https://phabricator.wikimedia.org/T195078) (owner: 10Rafidaslam) [11:27:18] I cannot see where hashar wanted to add that preference. Maybe it's only for elevated users as my preference/settings does not have any emoji field or something similar [11:35:26] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: In Gerrit some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196869#4271709 (10hashar) a:03Dzahn Fixed mutante ! Danke. My test case was to go to https://gerrit.wikimedia.org/r/settings/ and try to fi... [11:41:09] 10Phabricator, 10Developer-Relations (Apr-Jun-2018), 10Patch-For-Review: Try to identify new developers (via assignee field) in Phab tasks and potentially follow up - https://phabricator.wikimedia.org/T195780#4271733 (10Aklapper) 05Open>03Resolved Thanks Dzahn (for patience and reviewing and merging)! (... [11:52:30] 10Project-Admins: Rename Wikimedia project tags in Phab (#Wikisource; #Wiktionary) to clarify that they are not about language specific instances? - https://phabricator.wikimedia.org/T196740#4267022 (10Nemo_bis) I'm not sure what's wrong with subdomain-specific tasks. The problem of the tasks you linked is that... [11:58:58] (03CR) 10Hashar: [V: 032 C: 032] Mark repository as read only [extensions/DidYouMean] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439300 (https://phabricator.wikimedia.org/T196430) (owner: 10MarcoAurelio) [11:59:28] !log github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-DidYouMean | T196430 [11:59:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:59:31] T196430: Archive the DidYouMean extension - https://phabricator.wikimedia.org/T196430 [12:00:13] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4271771 (10hashar) [12:03:05] 10Project-Admins: Rename Wikimedia project tags in Phab (#Wikisource; #Wiktionary) to clarify that they are not about language specific instances? - https://phabricator.wikimedia.org/T196740#4271784 (10Aklapper) >>! In T196740#4271757, @Nemo_bis wrote: > I'm not sure what's wrong with subdomain-specific tasks. N... [12:03:39] 10Phabricator, 10Project-Admins, 10Pywikibot-core: Move #Pywikibot-Other-scripts to #Pywikibot-Scripts and transform it into parent project - https://phabricator.wikimedia.org/T195855#4271788 (10Dvorapa) [12:03:58] 10Phabricator, 10Project-Admins, 10Pywikibot-core: Move #Pywikibot-Other-scripts to #Pywikibot-Scripts and transform it into parent project - https://phabricator.wikimedia.org/T195855#4239337 (10Dvorapa) [12:08:29] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: In Gerrit some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196869#4271807 (10Dzahn) 05Open>03Resolved [12:24:10] 10Gerrit, 10Phabricator: "Update patch set 1" - https://phabricator.wikimedia.org/T196891#4271836 (10Tgr) [12:25:43] 10Gerrit, 10Phabricator: "Update patch set 1" - https://phabricator.wikimedia.org/T196891#4271852 (10Tgr) [12:25:56] 10Gerrit, 10Phabricator: "Update patch set 1" - https://phabricator.wikimedia.org/T196891#4271836 (10Paladox) Yep, we have a change ready to filter this ref. This ref is notedb so this is what it would look like in the db. [12:26:29] 10Gerrit, 10Phabricator: "Update patch set 1" - https://phabricator.wikimedia.org/T196891#4271855 (10Paladox) [12:26:41] 10Gerrit, 10Phabricator: "Update patch set 1" - https://phabricator.wikimedia.org/T196891#4271856 (10Tgr) Great, thanks! [12:27:58] 10Gerrit, 10Phabricator: "Update patch set 1" - https://phabricator.wikimedia.org/T196891#4271861 (10Paladox) The only thing is this https://phabricator.wikimedia.org/rERLSab649c6a5254e39dc1bef4a53eb3bcaee29f0e6b does not have a branch. So we would need to set the repos to not publish / notify. [12:31:20] 10Diffusion, 10Striker: Diffusion repository creation fails via toolsadmin - https://phabricator.wikimedia.org/T182142#4271881 (101-Byte) It seems that the creation of new repositories failed again. A reload does not solve the issue: {F21744344} [12:45:40] no_justification new steps to fix external id's in notedb https://groups.google.com/forum/#!topic/repo-discuss/7fJ3TKpUvvA :) [12:46:48] 10Beta-Cluster-Infrastructure, 10Services (doing): Puppet failure on deployment-cpjobqueue - https://phabricator.wikimedia.org/T196829#4271898 (10Pchelolo) [12:47:24] 10Beta-Cluster-Infrastructure, 10Services (doing): Puppet failure on deployment-cpjobqueue - https://phabricator.wikimedia.org/T196829#4269648 (10Pchelolo) Seems like we need to set up a videoscaler in deployment-prep cc @Joe [12:50:56] well gwtui is not as easy as polygerrit to use find the dom heh. since in gwtui it loads far to early [13:01:38] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Jenkins: Jenkins does not work with Depends-On clause - https://phabricator.wikimedia.org/T196894#4271942 (10Dvorapa) [13:01:59] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Jenkins: Jenkins does not work with Depends-On clause - https://phabricator.wikimedia.org/T196894#4271952 (10Dvorapa) [13:02:51] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812#4269209 (10Tgr) The lack of editing support is a pretty serious limitation IMO, especially for beginners who might find that a much easier way of creating a pa... [13:03:31] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812#4271959 (10Paladox) Yeh, upstream have added that support in 2.16 / 3.0. [13:04:38] 10Gerrit: PolyGerrit doesn't support old UI URLs - https://phabricator.wikimedia.org/T196895#4271961 (10Niedzielski) [13:08:30] 10Gerrit: PolyGerrit doesn't support old UI URLs - https://phabricator.wikimedia.org/T196895#4271961 (10Paladox) Forwarded this upstream https://bugs.chromium.org/p/gerrit/issues/detail?id=9231 Seems this issue was fixed on the master branch some how. [13:12:54] 10Gerrit: PolyGerrit doesn't support old UI URLs - https://phabricator.wikimedia.org/T196895#4271985 (10Niedzielski) Thanks @Paladox! {icon thumbs-up} [13:18:46] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:35:06] 10Gerrit, 10Upstream: PolyGerrit doesn't support old UI URLs - https://phabricator.wikimedia.org/T196895#4272074 (10Aklapper) [13:46:01] 10Diffusion, 10Gerrit: Recent "Update patch set 1" notifications from Diffusion - https://phabricator.wikimedia.org/T196891#4272106 (10Aklapper) [13:47:27] 10Gerrit: public key error when fetching from Wikimedia remotes - https://phabricator.wikimedia.org/T196898#4272110 (10Paladox) [13:52:07] 10Gerrit: public key error when fetching from Wikimedia remotes - https://phabricator.wikimedia.org/T196898#4272145 (10Ebe123) Yes. [13:53:12] 10Gerrit: public key error when fetching from Wikimedia remotes - https://phabricator.wikimedia.org/T196898#4272148 (10Paladox) Can you try saving in your preference to see if this is not a case of T196869 [13:58:17] 10Gerrit: public key error when fetching from Wikimedia remotes - https://phabricator.wikimedia.org/T196898#4272172 (10Ebe123) Can't save my preferences as my ssh key is already there (and has worked for Gerrit in the past) Le lun. 11 juin 2018, à 10 h 53, Paladox a écrit :... [13:59:09] 10Gerrit: public key error when fetching from Wikimedia remotes - https://phabricator.wikimedia.org/T196898#4272175 (10Paladox) I ment anything in the preference :). [14:01:33] 10Gerrit: public key error when fetching from Wikimedia remotes - https://phabricator.wikimedia.org/T196898#4272192 (10Paladox) could you try ssh -p 29418 Ebe123@gerrit.wikimedia.org -vvv [14:02:02] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#4272193 (10Andrew) a:05Andrew>03None This is looking like it's outside my area of influence... [14:02:20] 10Beta-Cluster-Infrastructure, 10Services (doing): Puppet failure on deployment-cpjobqueue - https://phabricator.wikimedia.org/T196829#4272195 (10Joe) Given how the jobrunner code is now structured, just adding the same hosts working as jobrunners will do the trick, probably. [14:07:45] 10Gerrit: public key error when fetching from Wikimedia remotes - https://phabricator.wikimedia.org/T196898#4272220 (10Ebe123) Thank you, it seemed to be a T183205 problem; the ssh only worked with lowercase `ebe123`. Le lun. 11 juin 2018, à 11 h 01, Paladox a écrit : > Pal... [14:08:23] 10Gerrit: public key error when fetching from Wikimedia remotes - https://phabricator.wikimedia.org/T196898#4272233 (10Paladox) 05Open>03declined Per ^^ [14:11:36] (03PS2) 10Thiemo Kreuz (WMDE): Don't warn on @license in "namespace … { … }" scopes [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/438031 [14:12:27] (03CR) 10jerkins-bot: [V: 04-1] Don't warn on @license in "namespace … { … }" scopes [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/438031 (owner: 10Thiemo Kreuz (WMDE)) [14:36:12] 10Release-Engineering-Team, 10Scap, 10ORES, 10Scoring-platform-team (Current): ORES deployments blocked by mysterious tin.eqiad.wmnet error - https://phabricator.wikimedia.org/T196663#4272342 (10thcipriani) [14:36:14] 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap3 to 3.8.2-1 - https://phabricator.wikimedia.org/T196710#4272339 (10thcipriani) 05Open>03Resolved a:03fgiunchedi New package was uploaded and puppet should be setting it up on targets with the next run. [14:37:41] 10Release-Engineering-Team, 10Scap, 10ORES, 10Scoring-platform-team (Current): ORES deployments blocked by mysterious tin.eqiad.wmnet error - https://phabricator.wikimedia.org/T196663#4264985 (10thcipriani) New scap package was built and the version bump merged in puppet. The new version should be on all t... [14:46:01] 10Release-Engineering-Team, 10Scap, 10ORES, 10Scoring-platform-team (Current): ORES deployments blocked by mysterious tin.eqiad.wmnet error - https://phabricator.wikimedia.org/T196663#4272360 (10awight) @thcipriani Thanks! I'll retry our deployment in tonight's window and will report back either way. [14:47:41] Hi, I have a patch that refuses to be merged for unknown reasons :\ RoanKattouw has +2'ed it twice, and there's no action from the system both times :( anyone can help figuring it out ? https://gerrit.wikimedia.org/r/#/c/oojs/ui/+/244592/ [14:47:56] 10Phabricator: Private space for tracking internal security team activities - https://phabricator.wikimedia.org/T196542#4272378 (10chasemp) p:05Triage>03Normal [14:47:59] We aren't even getting any failures or anything. It just doesn't even start the process after +2 [14:48:42] 10Phabricator, 10Patch-For-Review: Phabricator footer links should use Special:MyLanguage for links that support it - https://phabricator.wikimedia.org/T196836#4272383 (10Aklapper) p:05Triage>03Lowest [14:53:59] mooeypoo: looking at the zuul logs for that change I see "Change is needed but can not be merged" which I think means that zuul wants https://gerrit.wikimedia.org/r/c/oojs/ui/+/437960 to merge before 244592 [14:55:11] 10Phabricator: Private space for tracking internal security team activities - https://phabricator.wikimedia.org/T196542#4272429 (10chasemp) 05Open>03Resolved a:03chasemp A space for this type of work does exist but was more narrowly scoped at the time. All the stakeholders there have since left. Instead... [14:55:52] polygerrit shows relationship chain [14:55:56] thcipriani: omg I completely didn't notice it's a chain [14:56:06] showing https://gerrit.wikimedia.org/r/c/oojs/ui/+/437960/6 as before https://gerrit.wikimedia.org/r/c/oojs/ui/+/244592/8 [14:56:20] 10Phabricator: Private space for tracking internal security team activities - https://phabricator.wikimedia.org/T196542#4272442 (10Aklapper) Ooooh, true. Thank you, @chasemp! Does https://www.mediawiki.org/wiki/Phabricator/Spaces welcome an update for S11? [14:56:28] thcipriani: yowza, thank you! it actually should be a chain, I thought the previous one was merged. Bah. Thanks! [14:56:43] mooeypoo: sure thing! glad I could help :) [14:57:07] thcipriani: just out of curiosity -- why can't we see a visual response to that? I was concerned that on my end, it seemed like *nothing* happened [14:58:12] thcipriani: Neither RoanKattouw nor I noticed it's a chain since it didn't start as one (as you can see, we tried to +2 it twice) -- would be beneficial if a comment could be made or something? similar to "can't merge because of rebase" sort of thing? [14:58:16] that's a good quesiton. I've never looked to see if zuul could comment about those issues. Having nothing happen in the patchset is not a helpful response. [14:58:23] 10Gerrit: public key error when fetching from Wikimedia remotes (due to uppercase user name) - https://phabricator.wikimedia.org/T196898#4272449 (10Aklapper) [14:58:45] thcipriani: yeah. Well, if it's possible, it would help. Thanks anyways for this! I'll go bug people to review the underlying commit ;) [14:58:59] :) [14:59:47] definitely, I'll take a look to see if zuul offers any hooks to comment on these kinds of merge problems [15:08:28] 10Continuous-Integration-Infrastructure: zuul should comment on gerrit when it fails to enqueue patches due to patch-chains - https://phabricator.wikimedia.org/T196910#4272478 (10thcipriani) [15:17:35] hi, is there any gerrit admin's around to be able to look in the logs for why https://gerrit.wikimedia.org/r/c/415384 is 500 [15:17:44] reported by bstorm [15:18:10] she suspects that is the change that was accidentially deleted a while back [15:21:13] https://gerrit.wikimedia.org/r/c/operations/puppet/+/415384 [15:36:05] hrm, the earliest mention of that in the error logs is from 2018-06-08: WARN com.google.gerrit.server.change.ChangeJson : Omitting corrupt change 415384 from results [15:36:27] I don't know what that means [15:37:40] i am guessing that was a draft change [15:37:44] g [15:37:48] https://groups.google.com/forum/#!topic/repo-discuss/nfD-4xpvLgY [15:38:31] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812#4269209 (10EBernhardson) While I'm not a huge fan of forcing a new ui, if this ui is going away in future releases I agree it's better to default to the new on... [15:39:18] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#4272602 (10Dzahn) Still looks like an LDAP issue as in T138672#4259771 [15:39:32] thcipriani does it mention something like "missing current patch set for change 290774" [15:42:06] hrm, not that I can see. Tried variations of: grep 415384 /var/lib/gerrit2/review_site/logs/error_log | grep -c -i missing [15:42:10] all come back 0 [15:42:41] ok [15:42:53] thcipriani does this 2018-06-08: WARN com.google.gerrit.server.change.ChangeJson : Omitting corrupt change 415384 from results have a full log? [15:43:00] apparently the issue i linked to is from 2015 [15:43:07] and was fixed a long time ago too [15:49:52] here's the stacktrace from it: https://phabricator.wikimedia.org/P7244 [15:50:03] missing object...someplace I guess [15:50:20] 10Release-Engineering-Team, 10Operations: Phabricator outbound email seems to have a SPOF of mx1001 - https://phabricator.wikimedia.org/T196916#4272661 (10herron) [15:50:30] 10Release-Engineering-Team, 10Operations: Phabricator outbound email seems to have a SPOF of mx1001 - https://phabricator.wikimedia.org/T196916#4272671 (10herron) What does the phabricator outbound mail config look like today? Do we already have both mx1001 and mx2001 configured as outbound mail servers? [15:51:06] thcipriani: I thanks I think that confirms that was the change that was deleted by mistake [15:51:08] https://phabricator.wikimedia.org/rOPUP1173c3f63c65f33ac0d063cc22c572b5f95d8244 [15:52:08] https://gerrit.wikimedia.org/r/#/q/Ic5fa53348577756c0700df717f6b06b9d8de9cde [16:09:36] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:29:10] 10Phabricator, 10Release-Engineering-Team, 10Mail, 10Operations: Phabricator outbound email seems to have a SPOF of mx1001 - https://phabricator.wikimedia.org/T196916#4272884 (10Aklapper) https://phabricator.wikimedia.org/config/all lists for `phpmailer.smtp-host` the value `mx1001.wikimedia.org;mx2001.wik... [16:30:00] 10Release-Engineering-Team (Kanban), 10Scap: Automate weekly branch cut - https://phabricator.wikimedia.org/T196517#4259349 (10hashar) Maybe we can run it from releases1001 which is isolated from CI. Though the box is serving live traffic and the Jenkins instance is public https://releases-jenkins.wikimedia.org/ [16:39:13] 10Gerrit, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#4272917 (10Jdlrobson) 😋😃 [16:44:42] * paladox needs someone to review / merge / deploy https://phabricator.wikimedia.org/D1067 :) [16:44:51] since phabricator has high cpu load [16:45:37] twentyafterfour around for ^^? :) [16:49:42] paladox: hashar accepted the Diff but I guess thats != +2 and something else should be done :) [16:50:07] I don't know the next stepts of Differential [16:50:08] Hauskatze yeh, someone needs to land it (which i doin't have the rights to do :)) [16:50:22] arc land? or any interface button? [16:50:33] arc land [16:50:55] hate Differential then if you cannot just merge from the web UI as we do on Gerrit [16:51:25] :) [16:51:48] heh [16:51:49] * Hauskatze doesn't have Arcanist [16:52:00] you can technolly do it but that requires advanced setup [16:52:58] aha [16:53:43] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests: Add thcipriani and hashar to gerrit-root - https://phabricator.wikimedia.org/T196702#4272969 (10Dzahn) a:03Dzahn has been approved in SRE meeting [16:53:52] Hauskatze: mukunda will land and deploy [16:54:07] thanks greg-g :) [16:54:08] greg-g: awesome [16:56:29] Hauskatze: Yeah, there's no Web UI in Diffusion for merging patches. :-( [16:57:27] well, few things are impossible in informatics I was told :) so maybe anyone could code such a feature in the future [17:13:18] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Add thcipriani and hashar to gerrit-root - https://phabricator.wikimedia.org/T196702#4266006 (10herron) This was approved in the SRE meeting. Moving forward with the patch now. [17:14:05] thcipriani: I found a cool thing today! https://github.com/axkibe/lsyncd [17:15:50] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Add thcipriani to phabricator-roots - https://phabricator.wikimedia.org/T196703#4266022 (10herron) This was approved in the SRE meeting. Moving forward with the patch now. [17:16:12] no_justification: interesting [17:17:55] lua [17:20:54] paladox: landed, needs deployment now? [17:21:02] Hauskatze yep [17:21:56] ok :) [17:36:31] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Add thcipriani and hashar to gerrit-root - https://phabricator.wikimedia.org/T196702#4273119 (10herron) 05Open>03Resolved Access granted! ``` cobalt:~$ id hashar uid=1010(hashar) gid=500(wikidev) groups=500(w... [17:38:34] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Add thcipriani and hashar to gerrit-root - https://phabricator.wikimedia.org/T196702#4266006 (10thcipriani) Awesome, thanks @Dzahn, access looks good to me! [17:39:02] congrats thcipriani - now you can delete gerrit altogether :P [17:40:11] I can but I won't: this is the the test of one's ability to wield root access [17:40:25] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Add thcipriani to phabricator-roots - https://phabricator.wikimedia.org/T196703#4273125 (10herron) 05Open>03Resolved a:03herron All set! ``` phab1001:~$ id thcipriani uid=11634(thcipriani) gid=500(wikidev)... [17:44:24] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Add thcipriani and hashar to gerrit-root - https://phabricator.wikimedia.org/T196702#4266006 (10hashar) Works for me as well. Thank you @herron and @Dzahn [17:54:17] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812#4273184 (10Paladox) I am thinking about doing this next monday (which will give users plenty of time to give there opinion). I will update the email with the... [17:59:00] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4273195 (10mmodell) I'm going to stop phd and attempt to clear out the backlog from the queue (it's a lot of useless updates that we don't need to write to the db ultimately) [18:00:24] paladox: the one thing I'm missing in polygerrit is that the change table only shows +1/-1/checkmark, it doesn't show who left those votes [18:00:38] legoktm hover over them :) [18:01:05] is there any way to show them by default? I like being able to see all of them at once [18:02:12] legoktm on the status:open page or change screen? [18:03:00] status:open page [18:03:03] ok [18:03:06] (and any other search query/dashboard) [18:03:30] legoktm if you hover over them it will show you who left them. [18:03:39] right, I just don't want to have to do that [18:03:43] ah [18:03:44] hmm [18:04:06] old UI ftw [18:04:08] i doin't think there is a way to show them by default [18:04:09] :) [18:04:25] old ui you have to hover too :) [18:09:07] (03PS2) 10Legoktm: zuul: Add Chicocvenancio to jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/436957 (owner: 10Chico Venancio) [18:09:14] (03CR) 10Legoktm: [C: 032] zuul: Add Chicocvenancio to jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/436957 (owner: 10Chico Venancio) [18:10:48] (03CR) 10Legoktm: [C: 032] Whitelist Igor Leturia Azkarate [integration/config] - 10https://gerrit.wikimedia.org/r/439547 (owner: 10Sebastian Berlin (WMSE)) [18:10:58] (03Merged) 10jenkins-bot: zuul: Add Chicocvenancio to jenkins whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/436957 (owner: 10Chico Venancio) [18:12:55] (03Merged) 10jenkins-bot: Whitelist Igor Leturia Azkarate [integration/config] - 10https://gerrit.wikimedia.org/r/439547 (owner: 10Sebastian Berlin (WMSE)) [18:13:41] (03PS1) 10Paladox: Allow registered users to merge there own changes [All-Avatars] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439642 [18:13:51] !log deployed https://gerrit.wikimedia.org/r/436957 https://gerrit.wikimedia.org/r/439547 [18:13:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:26:44] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic, 10MW-1.31-release: Expand the set of bundled extensions and skins in MediaWiki 1.31 - https://phabricator.wikimedia.org/T196648#4273302 (10Legoktm) 05Open>03Resolved a:03CCicalese_WMF Yep! While we didn't get everything bundled that we wanted to,... [18:26:47] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10Epic: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4273305 (10Legoktm) [18:32:28] mwext-mw-selenium-composer-jessie is making trouble again... wasn't it made non-voting some time ago? [18:49:39] Hauskatze: Is https://phabricator.wikimedia.org/T191374 something you can do? [18:51:14] Should it need archiving? [18:51:19] Just change it to a phab replica? [18:51:38] No idea. [18:53:56] Just working through old e-mails. [18:56:50] 10Phabricator, 10Release-Engineering-Team, 10Mail, 10Operations: Phabricator outbound email seems to have a SPOF of mx1001 - https://phabricator.wikimedia.org/T196916#4272661 (10Reedy) >>! In T196916#4272671, @herron wrote: > What does the phabricator outbound mail config look like today? > > Do we alrea... [18:56:54] James_F: sorry, I was with the coroner at the phone, looking [18:57:29] well, if it is just moving the diffusion to gerrit I think I can do most of it yes [19:05:55] I see the content was already migrated so I can convert that to observe gerrit [19:06:10] asked on the task though before doing anything [19:08:14] got to go, ping me on phab if you need something James_F :) [19:29:26] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Build base image for math extension pipeline tests - https://phabricator.wikimedia.org/T196939#4273473 (10thcipriani) p:05Triage>03Normal [19:33:51] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Build base image for math extension pipeline tests - https://phabricator.wikimedia.org/T196939#4273486 (10thcipriani) [19:33:53] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Build Math extension container on Postmerge - https://phabricator.wikimedia.org/T196414#4273487 (10thcipriani) 1. We want a base image with apache and php but not MediaWiki * https://phabricator.wikimedia.org/P7246 * https://github.com/wikimedia/inte... [19:43:23] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Establish shared library for pipeline code used in Jenkins - https://phabricator.wikimedia.org/T196940#4273492 (10thcipriani) p:05Triage>03Normal [19:48:14] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Create integration/config pipeline job to create runnable MediaWiki + Math image - https://phabricator.wikimedia.org/T196944#4273557 (10thcipriani) p:05Triage>03Normal [19:50:12] I think it might be time for phabricator to ignore refs/changes [19:50:23] gerrit is absolutely overwhelming phabricator with junk [19:52:07] 10Phabricator, 10Release-Engineering-Team, 10Mail, 10Operations, 10Wikimedia-Incident: Phabricator outbound email seems to have a SPOF of mx1001 - https://phabricator.wikimedia.org/T196916#4273569 (10herron) [19:53:43] twentyafterfour it would be parsing the existing refs/changes/*/*/meta that it imported though it shoulden't be importing any more of those. [20:01:57] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Establish shared library for pipeline code used in Jenkins - https://phabricator.wikimedia.org/T196940#4273618 (10thcipriani) [20:09:49] PROBLEM - SSH on deployment-deploy-01 is CRITICAL: Connection refused [20:12:49] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#4273661 (10SMcCandlish) If it comes down to it, I'm okay with nuking SMcCandlish, smccandlish, a... [20:16:25] paladox: there are other refs that it's parsing. 8 million of them [20:18:42] 8M?! [20:18:54] geez that's a lot of time... [20:22:21] 8million wow [20:22:26] twentyafterfour which other ref? [20:23:03] twentyafterfour i guess it's parsing the existing refs [20:23:13] as all my patch does is prevent it cloning any more of those refs [20:29:44] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review, 10Prometheus-metrics-monitoring, 10User-fgiunchedi: Move deployment-prep redis instances to stretch - https://phabricator.wikimedia.org/T179371#4273745 (10Krenair) Alright. Leaving open pending deletion of the old redis hosts in a few weeks... [20:43:51] PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:45:06] it's still importing refs//meta things, is that expected? [20:45:39] yep [20:45:42] refs/meta/config [20:45:45] https://phabricator.wikimedia.org/R1889:b7f9d95daf8cc5dfadff9a8b92e10a3f0f4f842a <-- cf. from 7 Feb. 2017 [20:45:59] oh [20:46:00] thats refs/*/*/meta, not refs/meta/config [20:46:09] it should not be importing any new of those refs [20:46:10] appeared just now in the Phab feed [20:46:13] existing ones will show though [20:46:23] the ones it cloned before we deployed that change [20:46:30] twentyafterfour: https://phabricator.wikimedia.org/R1889:b7f9d95daf8cc5dfadff9a8b92e10a3f0f4f842a <-- expected to continue happening? [20:48:46] yes that's expected [20:48:49] for existing ones [20:49:02] so it should begin to die down (the load in the next few days) [20:49:16] as the ref is now in the repo on disk. [20:53:08] maybe those could be deleted or not counted after this is done... [20:53:14] idk if that'd be possible [21:05:03] 10Beta-Cluster-Infrastructure, 10Collaboration-Team-Triage: [betalabs]: Page triage: "Uncaught TypeError: Cannot read property 'getLogPageTitle' of undefined" for 'Redirects for discussion' - https://phabricator.wikimedia.org/T196954#4273887 (10Etonkovidova) [21:08:54] 10Beta-Cluster-Infrastructure, 10Collaboration-Team-Triage, 10MediaWiki-extensions-PageCuration: [betalabs]: Page triage: "Uncaught TypeError: Cannot read property 'getLogPageTitle' of undefined" for 'Redirects for discussion' - https://phabricator.wikimedia.org/T196954#4273887 (10Etonkovidova) [21:20:08] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4273955 (10Paladox) Apparently in gwtui it loads the js file to early to be able to find the dom like polygerrit, anyways we are switching polygerrit to be the default ui for new... [21:23:52] greg-g: I missed the Services window due to time zones, but am still itching to do a benign deployment tonight, in order to test the new scap version (and its git-lfs/submodule behavior). [21:24:20] Shall I book myself 21:30-22:00, maybe? [21:25:27] oops, I misread the calendar--no slots tonight, I'll wait until tomorrow's regular window. [21:26:02] * awight collides with antimatter for the day [21:30:22] awight: if it's just the git-lfs testing you can do it now, I don't think the Security team has anything happening [21:31:27] cool, I will then. It's a routine deployment, with some new models for ORES, I was going to test the canary but then roll ahead with the full cluster if it looks good. [21:31:51] o/ awight [21:32:01] halfak: :) [21:32:16] sorry to mess with your deploy changesets. I'd be happy to get that cleaned up with you now or wait until tomorrow. [21:32:54] maybe we should wait so you can get a clean scap test [21:32:57] awight, ^ [21:33:46] halfak: I think we'd be fine either merging my or your changes [21:34:00] I tested my big one and it all works together. [21:34:07] I think some of your changes have issues. [21:34:24] I'll go with yours then, thanks for testing. [21:43:26] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:02:31] twentyafterfour if we manage to remove these references from the repos and delete them from the db (so they are not index by phd) that should fix it. [22:20:09] 10Continuous-Integration-Config, 10MediaWiki-Docker, 10Readers-Web-Backlog (Tracking): mwext-MobileFrontend-npm-run-lint-modules-docker failing - node script running as php - https://phabricator.wikimedia.org/T196956#4274042 (10Jdlrobson) [22:23:11] it seems it stopped importing that ref now twentyafterfour [22:23:18] it's now parsing the existing ones :) [22:23:25] which i am not sure how to stop [22:44:59] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T191054#4274102 (10greg) a:03dduvall [22:45:28] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T191055#4274103 (10greg) 05Open>03declined No Train this week due to the SRE summit. [22:46:01] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T191056#4274106 (10greg) a:03dduvall [22:49:20] thcipriani wondering could you review https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/439503/ please? :) [22:49:58] polygerrit patch to add a footer with CoC and privacy policy [23:08:36] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:18:18] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:41:28] 10Beta-Cluster-Infrastructure, 10Operations: Mails through deployment-mx SPF & DKIM fails - https://phabricator.wikimedia.org/T87338#4274206 (10Krenair) Alright so I had some run-ins with Designate while trying to do DKIM (turns out you can't use a 2048 bit RSA key because that puts your public key over a leng... [23:54:59] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]