[00:49:27] Project performance-webpagetest-wpt-org build #1873: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/performance-webpagetest-wpt-org/1873/ [04:18:05] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #132: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/132/ [04:50:57] Yippee, build fixed! [04:50:57] Project performance-webpagetest-wpt-org build #1874: 09FIXED in 18 min: https://integration.wikimedia.org/ci/job/performance-webpagetest-wpt-org/1874/ [07:11:40] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:12:44] PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:17:50] PROBLEM - Puppet run on deployment-zookeeper01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:43:29] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:45:24] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:57:48] RECOVERY - Puppet run on deployment-zookeeper01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:03:29] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [08:15:24] RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [08:17:44] RECOVERY - Puppet run on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [08:21:38] RECOVERY - Puppet run on integration-slave-precise-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [08:23:58] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:24:38] 10Beta-Cluster-Infrastructure, 06Operations, 05Prometheus-metrics-monitoring: deploy prometheus node_exporter and server to deployment-prep - https://phabricator.wikimedia.org/T144502#2608192 (10hashar) [08:24:40] 10Beta-Cluster-Infrastructure, 07Tracking: Setup monitoring for Beta Cluster (tracking) - https://phabricator.wikimedia.org/T53497#2608191 (10hashar) [08:25:15] 10Beta-Cluster-Infrastructure, 07Tracking: Setup monitoring for Beta Cluster (tracking) - https://phabricator.wikimedia.org/T53497#581090 (10hashar) [08:25:18] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner: Beta cluster job queue is unmonitored / potentially not running all jobs - https://phabricator.wikimedia.org/T128357#2608194 (10hashar) [08:25:25] 10Beta-Cluster-Infrastructure, 10MediaWiki-JobRunner, 10Monitoring: Beta cluster job queue is unmonitored / potentially not running all jobs - https://phabricator.wikimedia.org/T128357#2071532 (10hashar) [08:25:37] 10Beta-Cluster-Infrastructure, 10Monitoring, 07Tracking: Setup monitoring for Beta Cluster (tracking) - https://phabricator.wikimedia.org/T53497#581090 (10hashar) [08:38:58] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:00:52] RECOVERY - Puppet run on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [09:13:01] RECOVERY - Puppet run on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [09:14:03] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW: Remove https://integration.wikimedia.org/ci/job/selenium-Wikidata/ - https://phabricator.wikimedia.org/T144487#2608344 (10Tobi_WMDE_SW) @zeljkofilipin If nobody was assigned before a task is always assigned to the one who... [09:15:11] RECOVERY - Puppet run on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [09:16:35] RECOVERY - Puppet run on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [09:21:28] PROBLEM - Puppet staleness on deployment-salt02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0] [10:15:43] (03CR) 10Hashar: [C: 032] Archive Extension:ApiSandbox on ZUUL [integration/config] - 10https://gerrit.wikimedia.org/r/308325 (https://phabricator.wikimedia.org/T127012) (owner: 10MarcoAurelio) [10:16:01] (03CR) 10Hashar: "Lets add it back as a dummy entry. Can clear up all of them from CI later on as needed." [integration/config] - 10https://gerrit.wikimedia.org/r/308325 (https://phabricator.wikimedia.org/T127012) (owner: 10MarcoAurelio) [10:16:20] 10Continuous-Integration-Config, 13Patch-For-Review: Add "fail-archived-repositories" to commits to mediawiki/extensions/ApiSandbox in Gerrit - https://phabricator.wikimedia.org/T127012#2608422 (10hashar) 05Open>03Resolved a:03MarcoAurelio Marco fixed it :] [10:16:43] (03Merged) 10jenkins-bot: Archive Extension:ApiSandbox on ZUUL [integration/config] - 10https://gerrit.wikimedia.org/r/308325 (https://phabricator.wikimedia.org/T127012) (owner: 10MarcoAurelio) [10:26:42] (03CR) 10Hashar: [C: 04-1] "This is going to have unwanted side effects. Whenever a file from the polyfill is changed, the job is going to be skip entirely and thus " [integration/config] - 10https://gerrit.wikimedia.org/r/307033 (https://phabricator.wikimedia.org/T143598) (owner: 10Paladox) [10:27:01] 10Continuous-Integration-Config, 10Fundraising-Backlog, 13Patch-For-Review: symfony-polyfill54 is breaking CI - https://phabricator.wikimedia.org/T143598#2573044 (10hashar) The php55lint job in Jenkins is very lame and does not support any exclusion list. If that is needed, we should move to use the 'compo... [10:28:15] (03Abandoned) 10Hashar: Create a customised debian-glue test [integration/config] - 10https://gerrit.wikimedia.org/r/307067 (https://phabricator.wikimedia.org/T144094) (owner: 10Paladox) [10:30:07] (03Abandoned) 10Hashar: Revert "Temporarily move composer-hhvm/php5 jobs off of nodepool" [integration/config] - 10https://gerrit.wikimedia.org/r/304145 (owner: 10Paladox) [10:31:39] (03CR) 10Paladox: "@hashar I'm not sure if we can do composer since this is already a vendor folder. See https://phabricator.wikimedia.org/diffusion/WFCM/" [integration/config] - 10https://gerrit.wikimedia.org/r/307033 (https://phabricator.wikimedia.org/T143598) (owner: 10Paladox) [10:32:38] (03Abandoned) 10Paladox: Create wikimedia-fundraising-crm-vendor-php55lint test [integration/config] - 10https://gerrit.wikimedia.org/r/307033 (https://phabricator.wikimedia.org/T143598) (owner: 10Paladox) [10:33:14] (03CR) 10Paladox: "I guess we could do a noop job instead since we doint really need to lint a vendor folder? Should we do that @hashar?" [integration/config] - 10https://gerrit.wikimedia.org/r/307033 (https://phabricator.wikimedia.org/T143598) (owner: 10Paladox) [10:38:42] (03Abandoned) 10Hashar: Revert "Move mediawiki-core-phpcs off of nodepool" [integration/config] - 10https://gerrit.wikimedia.org/r/304150 (owner: 10Paladox) [10:39:33] (03Abandoned) 10Hashar: Testing [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/301361 (owner: 10Paladox) [10:41:41] 10Continuous-Integration-Config, 10Fundraising-Backlog, 13Patch-For-Review: symfony-polyfill54 is breaking CI - https://phabricator.wikimedia.org/T143598#2608460 (10Paladox) @awight or @Ejegg hi, could we use composer to do the phplint please? It woud look strange but it will allow us to continue linting th... [11:17:58] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2608540 (10Tobi_WMDE_SW) [11:31:21] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Jul-Sep-2016): Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2608569 (10Aklapper) [11:32:00] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2601243 (10Paladox) We carn't remove wikidata/browsertests.git but we can disable it. [12:16:04] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:23:38] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:26:15] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:39:18] (03Abandoned) 10Hashar: [AjaxLogin] Make extension unit tests non voting until tests are fixed [integration/config] - 10https://gerrit.wikimedia.org/r/303794 (owner: 10Paladox) [12:39:51] (03CR) 10Hashar: "Which is already tracked by T126415" [integration/config] - 10https://gerrit.wikimedia.org/r/303794 (owner: 10Paladox) [12:47:38] 06Release-Engineering-Team: Feedback for European SWAT window - https://phabricator.wikimedia.org/T143894#2608674 (10hashar) 05Open>03Resolved a:03hashar Thank you everyone for the feedback. There is no follow-up action, I was just being curious :] [12:51:05] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [13:01:14] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [13:03:40] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [13:11:28] RECOVERY - Puppet staleness on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [3600.0] [13:15:58] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2608737 (10Tobi_WMDE_SW) [13:47:38] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #136: 04FAILURE in 3 min 37 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/136/ [13:52:52] 10Continuous-Integration-Infrastructure, 07Puppet: Cant refresh Nodepool snapshot due to puppet: Could not find class passwords::puppet::database - https://phabricator.wikimedia.org/T143769#2608847 (10hashar) I think the original issue related to puppetmaster class is gone. The reason for Alexandros change is... [13:54:33] (03PS1) 10Hashar: dib: override standard / diamond::collector [integration/config] - 10https://gerrit.wikimedia.org/r/308567 (https://phabricator.wikimedia.org/T143769) [14:04:52] (03CR) 10Hashar: [C: 032] "That fixed the build of both Jessie and Trusty images. I have confirmed that neither ::standard or diamond are included." [integration/config] - 10https://gerrit.wikimedia.org/r/308567 (https://phabricator.wikimedia.org/T143769) (owner: 10Hashar) [14:05:07] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Puppet: Cant refresh Nodepool snapshot due to puppet: Could not find class passwords::puppet::database - https://phabricator.wikimedia.org/T143769#2608943 (10hashar) 05Open>03Resolved a:03hashar That fixed the build of both Jessie and Trust... [14:05:14] (03PS2) 10Hashar: dib: override standard / diamond::collector [integration/config] - 10https://gerrit.wikimedia.org/r/308567 (https://phabricator.wikimedia.org/T143769) [14:05:19] (03CR) 10Hashar: dib: override standard / diamond::collector [integration/config] - 10https://gerrit.wikimedia.org/r/308567 (https://phabricator.wikimedia.org/T143769) (owner: 10Hashar) [14:05:22] (03CR) 10Hashar: [C: 032] dib: override standard / diamond::collector [integration/config] - 10https://gerrit.wikimedia.org/r/308567 (https://phabricator.wikimedia.org/T143769) (owner: 10Hashar) [14:06:26] (03Merged) 10jenkins-bot: dib: override standard / diamond::collector [integration/config] - 10https://gerrit.wikimedia.org/r/308567 (https://phabricator.wikimedia.org/T143769) (owner: 10Hashar) [14:08:11] !log Refreshing Nodepool base images for Trusty and Jessie. Managed to build new ones after T143769 [14:08:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:29:18] {u'show': u'true'} is not of type 'string' [14:29:19] yeah [14:29:24] ;( [14:30:11] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Puppet: Cant refresh Nodepool snapshot due to puppet: Could not find class passwords::puppet::database - https://phabricator.wikimedia.org/T143769#2609093 (10hashar) Cant refresh the snapshots since uploading the images fail with: BadReque... [14:31:34] RoanKattouw: Hi, i think i know how to fix T144565 [14:31:44] changing https://github.com/gerrit-review/gerrit/blob/7f4c48f37b9a8970ae2f8c8dc881f603ec9e707f/gerrit-gwtui/src/main/java/com/google/gerrit/client/diff/SideBySide.java#L254 to true [14:31:58] will work :), now i need to figure out how i can change it through js or gerrit config [14:34:20] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API - https://phabricator.wikimedia.org/T144753#2609115 (10hashar) [14:36:30] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609129 (10hashar) [14:46:07] 05Continuous-Integration-Scaling, 06Operations, 07Nodepool, 07WorkType-NewFunctionality: Backport python-shade from debian/testing to jessie-wikimedia - https://phabricator.wikimedia.org/T107267#2609155 (10hashar) **status update** I haven't come to it cause the next commit in Nodepool relying on it is q... [14:46:33] 05Continuous-Integration-Scaling, 06Operations, 07Nodepool, 07WorkType-NewFunctionality: Backport python-shade from debian/testing to jessie-wikimedia - https://phabricator.wikimedia.org/T107267#2609157 (10hashar) p:05Normal>03Low [14:50:10] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609168 (10hashar) On labnet1002 the API log shows the query has been done ag... [14:52:37] 03Scap3, 10Mathoid, 06Services, 15User-mobrovac: Enable Scap3 config deploys for Mathoid - https://phabricator.wikimedia.org/T144755#2609180 (10mobrovac) [14:52:45] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609195 (10hashar) [14:52:55] 03Scap3, 06Services, 10service-runner, 10service-template-node, and 2 others: Enable config deploys for service::node services - https://phabricator.wikimedia.org/T144542#2609198 (10mobrovac) [14:53:00] 03Scap3, 10Mathoid, 06Services, 15User-mobrovac: Enable Scap3 config deploys for Mathoid - https://phabricator.wikimedia.org/T144755#2609197 (10mobrovac) [14:59:50] hashar hi, did you see https://wikitech.wikimedia.org/wiki/How_to_setup_Gerrit_in_Labs :) [15:00:09] i finally built a doc on getting gerrit into labs with alot of help from mutante. [15:00:16] ostriches fixed it to run on labs :) [15:07:42] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:09:39] 03Scap3, 10Mathoid, 06Services, 13Patch-For-Review, 15User-mobrovac: Enable Scap3 config deploys for Mathoid - https://phabricator.wikimedia.org/T144755#2609253 (10mobrovac) p:05Triage>03Normal [15:11:07] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609258 (10hashar) I got rid of the images properties via `glance image-updat... [15:18:21] 10Gerrit, 07Upstream: Gerrit's new side-by-side diff screen sometimes cuts off the last few characters of a line - https://phabricator.wikimedia.org/T144565#2609270 (10Paladox) We will need to manually patch gerrit with the change, but once we do that, we can send it upstream for them to review :) [15:22:08] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609286 (10hashar) I have deleted the image and created a new one without usi... [15:23:57] (03PS1) 10Niedzielski: Limit performance jobs IRC notification channels [integration/config] - 10https://gerrit.wikimedia.org/r/308579 [15:26:00] (03CR) 10Niedzielski: "This isn't my repo so I didn't deploy and test the changes (which I just guessed at)." [integration/config] - 10https://gerrit.wikimedia.org/r/308579 (owner: 10Niedzielski) [15:28:53] * paladox my first time at editing java :) [15:40:14] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609345 (10hashar) And I eventually remembered that Nodepool does inject prop... [15:42:43] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:37] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2609357 (10hashar) a:03hashar [16:01:40] Project selenium-CentralNotice » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #138: 04FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/138/ [16:26:23] 10Beta-Cluster-Infrastructure, 07Easy: monitor unsigned salt keys - https://phabricator.wikimedia.org/T72862#2609427 (10Dzahn) still needed for production as well ? -> T143846 [17:17:06] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:32:05] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [18:06:44] PROBLEM - Host deployment-parsoid05 is DOWN: CRITICAL - Host Unreachable (10.68.16.120) [20:03:43] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:13:04] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:23:53] 06Release-Engineering-Team, 06Commons, 10MediaWiki-File-management, 06Multimedia, and 4 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#2609727 (10Tau) Hi! Finally InstantCommons is working in my wiki again. I don't know exactly what caused the error but after u... [20:24:35] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:43:42] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:45:42] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:47:36] 10Gerrit, 07Upstream: Gerrit's new side-by-side diff screen sometimes cuts off the last few characters of a line - https://phabricator.wikimedia.org/T144565#2609754 (10Paladox) Ok I have built the deb but it is gerrit 2.13, I will need @demon to pull in upstream 2.12 into our repo here for me to submit our cus... [20:53:05] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [21:04:36] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [21:16:05] PROBLEM - Puppet run on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:20:41] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:05] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:46:34] PROBLEM - Puppet run on deployment-parsoid09 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:46:34] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:46:42] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:46:54] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:47:24] Puppet errors ^^ [21:47:32] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:47:40] PROBLEM - Puppet run on deployment-poolcounter02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:51:10] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:51:46] PROBLEM - Puppet run on deployment-fluorine02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:51:58] PROBLEM - Puppet run on deployment-pdfrender is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:52:04] PROBLEM - Puppet run on deployment-zotero01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:52:33] PROBLEM - Puppet run on deployment-apertium01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:52:49] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:54:59] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:55:05] paladox, yes [21:55:13] those are indeed puppet errors [21:55:14] Ok [21:55:16] prod had a similar thing [21:55:20] it's been dealt with [21:55:20] Oh [21:55:23] thanks [21:55:34] well, prod had the exact same thing really [21:56:07] PROBLEM - Puppet run on deployment-db2 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:56:09] PROBLEM - Puppet run on deployment-pdf02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:56:13] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [21:56:32] Oh [21:56:47] PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:57:19] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:57:21] PROBLEM - Puppet run on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:57:31] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:57:57] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:01:10] PROBLEM - Puppet run on deployment-ircd is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:01:18] PROBLEM - Puppet run on deployment-memc04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:01:50] PROBLEM - Puppet run on deployment-imagescaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:01:56] PROBLEM - Puppet run on deployment-db03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:02:02] PROBLEM - Puppet run on deployment-sentry01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:02:04] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:04:31] 10Gerrit, 07Upstream: Gerrit's new side-by-side diff screen sometimes cuts off the last few characters of a line - https://phabricator.wikimedia.org/T144565#2609915 (10Paladox) @demon can we do a small customisation change to gerrit 2.12, to switch this default pref on by default please? [22:17:17] RECOVERY - Puppet run on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:17:31] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:21:35] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:21:41] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [22:24:05] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [22:26:07] RECOVERY - Puppet run on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [22:26:35] RECOVERY - Puppet run on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [22:26:47] RECOVERY - Puppet run on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:26:52] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [22:26:58] RECOVERY - Puppet run on deployment-pdfrender is OK: OK: Less than 1.00% above the threshold [0.0] [22:27:06] RECOVERY - Puppet run on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:27:34] RECOVERY - Puppet run on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:27:40] RECOVERY - Puppet run on deployment-poolcounter02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:27:50] RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [22:31:46] RECOVERY - Puppet run on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:35:01] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:36:09] RECOVERY - Puppet run on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [22:36:11] RECOVERY - Puppet run on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:36:13] RECOVERY - Puppet run on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [22:36:13] RECOVERY - Puppet run on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:36:51] RECOVERY - Puppet run on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:37:05] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:37:23] RECOVERY - Puppet run on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [22:37:31] RECOVERY - Puppet run on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:37:57] RECOVERY - Puppet run on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:18] RECOVERY - Puppet run on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:54] RECOVERY - Puppet run on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:42:00] RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:44:38] (03PS1) 10Andrew Bogott: Add a test for unlucky .pp files that break puppet runs. [integration/config] - 10https://gerrit.wikimedia.org/r/308668 (https://phabricator.wikimedia.org/T144774) [22:45:16] (03CR) 10jenkins-bot: [V: 04-1] Add a test for unlucky .pp files that break puppet runs. [integration/config] - 10https://gerrit.wikimedia.org/r/308668 (https://phabricator.wikimedia.org/T144774) (owner: 10Andrew Bogott) [22:47:01] (03PS2) 10Andrew Bogott: Add a test for unlucky .pp files that break puppet runs. [integration/config] - 10https://gerrit.wikimedia.org/r/308668 (https://phabricator.wikimedia.org/T144774) [22:47:39] (03CR) 10jenkins-bot: [V: 04-1] Add a test for unlucky .pp files that break puppet runs. [integration/config] - 10https://gerrit.wikimedia.org/r/308668 (https://phabricator.wikimedia.org/T144774) (owner: 10Andrew Bogott) [22:48:25] (03PS3) 10Andrew Bogott: Add a test for unlucky .pp files that break puppet runs. [integration/config] - 10https://gerrit.wikimedia.org/r/308668 (https://phabricator.wikimedia.org/T144774) [22:49:24] (03CR) 10jenkins-bot: [V: 04-1] Add a test for unlucky .pp files that break puppet runs. [integration/config] - 10https://gerrit.wikimedia.org/r/308668 (https://phabricator.wikimedia.org/T144774) (owner: 10Andrew Bogott) [22:51:02] (03Abandoned) 10Andrew Bogott: Add a test for unlucky .pp files that break puppet runs. [integration/config] - 10https://gerrit.wikimedia.org/r/308668 (https://phabricator.wikimedia.org/T144774) (owner: 10Andrew Bogott) [22:58:16] 10Gerrit, 07Upstream: Gerrit's new side-by-side diff screen sometimes cuts off the last few characters of a line - https://phabricator.wikimedia.org/T144565#2609974 (10Paladox) Partially fixed in https://gerrit-review.googlesource.com/#/c/85355/ we can create our own custom patch to set it true for line wrapping [22:58:43] (03PS1) 10Niedzielski: Remove Android Commons app job [integration/config] - 10https://gerrit.wikimedia.org/r/308669 [23:15:07] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:15:53] (03CR) 10Paladox: [C: 031] Remove Android Commons app job [integration/config] - 10https://gerrit.wikimedia.org/r/308669 (owner: 10Niedzielski) [23:27:16] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:50:04] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0]