[01:11:35] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T167536#3381044 (10Legoktm) [01:44:59] (03PS1) 10Chad: Drop symlink from tarball, hurts Windows users [tools/release] - 10https://gerrit.wikimedia.org/r/361611 (https://phabricator.wikimedia.org/T156491) [01:46:14] (03CR) 10Chad: [C: 032] Drop symlink from tarball, hurts Windows users [tools/release] - 10https://gerrit.wikimedia.org/r/361611 (https://phabricator.wikimedia.org/T156491) (owner: 10Chad) [01:46:51] (03Merged) 10jenkins-bot: Drop symlink from tarball, hurts Windows users [tools/release] - 10https://gerrit.wikimedia.org/r/361611 (https://phabricator.wikimedia.org/T156491) (owner: 10Chad) [01:49:44] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MediaWiki-Vendor, 10MW-1.28-release, and 2 others: Symbolic link in MediaWiki distribution causes problems for some Windows users - https://phabricator.wikimedia.org/T156491#3381078 (10demon) 05Open>03Resolved a:03demon >>! In T156491#2976... [01:51:25] 10Continuous-Integration-Config, 10RESTBase, 10Wikipedia-Android-App-Backlog, 10Services (watching): Kick off periodic Android CI tests when RESTBase is updated on beta labs - https://phabricator.wikimedia.org/T146488#3381083 (10Mholloway) 05Open>03declined We've moved away from integration-level tests... [03:15:42] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%) [03:15:48] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, and 6 others: Excluded vendor directory from lint tests when it is in an included extension - https://phabricator.wikimedia.org/T168597#3381116 (10thcipriani) [07:00:41] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [09:08:01] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, and 6 others: Excluded vendor directory from lint tests when it is in an included extension - https://phabricator.wikimedia.org/T168597#3381498 (10hashar) [09:12:37] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, and 6 others: Excluded vendor directory from lint tests when it is in an included extension - https://phabricator.wikimedia.org/T168597#3369431 (10hashar) p:05Triage>03Normal [09:19:40] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, and 6 others: Excluded vendor directory from lint tests when it is in an included extension - https://phabricator.wikimedia.org/T168597#3369431 (10hashar) The jenkins job roughly runs: composer update... [09:52:40] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:59:31] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:02:40] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [12:18:42] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:22:42] (03CR) 10Hashar: "I will redo that one in small chunks and reverify jobs still being on trusty." [integration/config] - 10https://gerrit.wikimedia.org/r/361496 (owner: 10Krinkle) [12:33:40] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [12:44:44] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:50:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intermittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3382045 (10hashar) Finally had a node... [13:00:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intermittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3382074 (10hashar) I have logged on t... [13:00:36] RECOVERY - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is OK: OK: Less than 100.00% above the threshold [0.0] [13:04:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Performance-Team, 10Jenkins, 10Upstream: WebPageTest job fails intermittently with "java.io.IOException: Unexpected termination of the channel" - https://phabricator.wikimedia.org/T166557#3382085 (10hashar) I have made the sl... [13:04:46] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:35:38] PROBLEM - Puppet errors on deployment-ms-be-test01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [13:36:10] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:37:30] the ms-be failures are me btw [13:37:39] testing T151648 [13:37:39] T151648: Implement storage policies for swift - https://phabricator.wikimedia.org/T151648 [13:37:49] no, testing T163673 [13:37:49] T163673: Some swift disks wrongly mounted on 5 ms-be hosts - https://phabricator.wikimedia.org/T163673 [13:38:21] 10Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 10Upstream: "Unhandled Exception ("Exception"): Rejecting ambiguous URI" when trying to access certain tasks / conpherence rooms / etc - https://phabricator.wikimedia.org/T166870#3382228 (10Aklapper) Still an issue, see T168942 [13:39:30] (03PS1) 10Aude: Update Wikidata to wmf/1.30.0-wmf.7 [tools/release] - 10https://gerrit.wikimedia.org/r/361666 [13:40:40] RECOVERY - Puppet errors on deployment-ms-be-test01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:47:04] Project selenium-VisualEditor ยป firefox,beta,Linux,BrowserTests build #441: 04FAILURE in 3 min 3 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/441/ [13:53:34] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:06:40] PROBLEM - Puppet errors on deployment-ms-be-test01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [14:11:10] RECOVERY - Puppet errors on deployment-ms-be03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:16:41] RECOVERY - Puppet errors on deployment-ms-be-test01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:35] RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:38:18] 10Gerrit, 10Differential, 10Repository-Admins: Separate Diffusion's rANWS from gerrit's analytics-wikistats - https://phabricator.wikimedia.org/T168549#3382414 (10Milimetric) [14:38:26] 10Gerrit, 10Differential, 10Repository-Admins: Separate Diffusion's rANWS from gerrit's analytics-wikistats - https://phabricator.wikimedia.org/T168549#3368089 (10Milimetric) Poke on this, thoughts? [15:11:12] 10MediaWiki-Codesniffer: Adjust MediaWiki.WhiteSpace.SpaceBeforeClassBrace.NoSpaceBeforeBrace for use with multiline class statements - https://phabricator.wikimedia.org/T168961#3382616 (10Umherirrender) [15:11:39] (03CR) 10Aude: [C: 032] Update Wikidata to wmf/1.30.0-wmf.7 [tools/release] - 10https://gerrit.wikimedia.org/r/361666 (owner: 10Aude) [15:12:15] (03CR) 10jerkins-bot: [V: 04-1] Update Wikidata to wmf/1.30.0-wmf.7 [tools/release] - 10https://gerrit.wikimedia.org/r/361666 (owner: 10Aude) [15:14:57] 10MediaWiki-Codesniffer: Check MediaWiki.Commenting.IllegalSingleLineComment.IllegalSingleLineCommentStart and Doxygen use - https://phabricator.wikimedia.org/T168963#3382645 (10Umherirrender) [15:15:32] 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-06-27_(1.30.0-wmf.7)), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T167535#3382663 (10MacFan4000) 05Open>03Resolved Seems to be successfully de... [15:24:37] hello, CI things look broken. i'm getting errors like this: [15:24:42] 15:16:01 >> [Tue Jun 27 15:16:00 2017] [hphp] [2098:7fdd0111c200:0:000001] [] Uncaught exception: Could not open extension /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory [15:24:57] examples: https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm-jessie/16610/console https://integration.wikimedia.org/ci/job/oojs-ui-npm-run-jenkins-node-6-jessie/776/console [15:25:17] greg-g: hashar: ^ [15:27:31] PROBLEM - Puppet errors on deployment-eventlogging03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:34:38] 10Gerrit, 10Differential, 10Repository-Admins: Separate Diffusion's rANWS from gerrit's analytics-wikistats - https://phabricator.wikimedia.org/T168549#3382729 (10mmodell) @Milimetric I started to make a ANW2 but numbers aren't allowed in callsigns. I separated the two so that they are not mirroring anymore... [15:37:07] MatmaRex i belive hashar fixed that today [15:37:27] though the nodepool images need updating still [15:37:40] https://gerrit.wikimedia.org/r/#/c/342635/5 [15:43:56] 10Gerrit, 10Differential, 10Repository-Admins: Separate Diffusion's rANWS from gerrit's analytics-wikistats - https://phabricator.wikimedia.org/T168549#3382758 (10mmodell) @milimetric: I've created {rWIKISTATS}. I have one question: Who should be able to push to this repo? [15:52:05] 10MediaWiki-Codesniffer: Add sniff to detect spaces before comma in argument lists - https://phabricator.wikimedia.org/T168970#3382818 (10Umherirrender) [16:15:46] 10Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 10Upstream: "Unhandled Exception ("Exception"): Rejecting ambiguous URI" when trying to access certain tasks / conpherence rooms / etc - https://phabricator.wikimedia.org/T166870#3382959 (10mmodell) @aklapper: I fear this will keep popping up in... [16:22:33] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Uncaught exception: Could not open extension when trying to load tidy.io - https://phabricator.wikimedia.org/T168978#3383021 (10Paladox) [16:22:48] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Uncaught exception: Could not open extension when trying to load tidy.io - https://phabricator.wikimedia.org/T168978#3383048 (10Paladox) p:05Triage>03Unbreak! [16:23:29] 10Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 10Upstream: "Unhandled Exception ("Exception"): Rejecting ambiguous URI" when trying to access certain tasks / conpherence rooms / etc - https://phabricator.wikimedia.org/T166870#3383057 (10mmodell) Ah, I see what happened - for some reason libph... [16:24:04] 10Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 10Upstream: "Unhandled Exception ("Exception"): Rejecting ambiguous URI" when trying to access certain tasks / conpherence rooms / etc - https://phabricator.wikimedia.org/T166870#3383076 (10mmodell) [16:30:18] 10Gerrit, 10Differential, 10Repository-Admins: Separate Diffusion's rANWS from gerrit's analytics-wikistats - https://phabricator.wikimedia.org/T168549#3383144 (10Milimetric) Thanks @mmodell. All analytics team members would be great, if they're in a group, if not: working on the repo: @Nuria @mforns @fdan... [16:33:09] 10Gerrit, 10Differential, 10Repository-Admins: Separate Diffusion's rANWS from gerrit's analytics-wikistats - https://phabricator.wikimedia.org/T168549#3383153 (10mmodell) @Milimetric The access is currently controlled by #acl_analytics. I've updated membership of that acl to reflect the people you mentioned. [16:33:18] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Differential, 10Repository-Admins: Separate Diffusion's rANWS from gerrit's analytics-wikistats - https://phabricator.wikimedia.org/T168549#3383156 (10mmodell) 05Open>03Resolved a:03mmodell [16:35:02] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Differential, 10Repository-Admins: Separate Diffusion's rANWS from gerrit's analytics-wikistats - https://phabricator.wikimedia.org/T168549#3383164 (10Milimetric) Thanks very much, you rock. [16:36:01] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Differential, 10Repository-Admins: Separate Diffusion's rANWS from gerrit's analytics-wikistats - https://phabricator.wikimedia.org/T168549#3383167 (10mmodell) @milimetric: Also, fyi - any member of the analytics team should be able to edit the repo in diffu... [16:55:17] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10WorkType-NewFunctionality: Configure phabricator clustering for daemons and repositories - https://phabricator.wikimedia.org/T143175#3383284 (10mmodell) >>! In T143175#3374349, @Volans wrote: > Is not fully clear to me right now in the... [17:07:51] 10MediaWiki-Codesniffer: Add sniff to check if all returns of a function are void or non-void returns - https://phabricator.wikimedia.org/T168994#3383413 (10Umherirrender) [17:08:01] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:08:22] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog, 10Reading-Web-Kanban-Board: Create bot that automatically rebases and rebuilds patches to master - https://phabricator.wikimedia.org/T167181#3383427 (10Jdlrobson) a:05Jdlrobson>03Jhernandez [17:39:07] !log running mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type 'external-id' --property-id P34 [17:39:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:46:00] 10Continuous-Integration-Config, 10Release-Engineering-Team (Watching / External), 10Discovery, 10Discovery-Analysis (Current work): Add lint/CI to all wikimedia/discovery analytics repositories - https://phabricator.wikimedia.org/T153856#3383673 (10mpopov) >>! In T153856#3375157, @hashar wrote: > The summ... [17:46:11] 10MediaWiki-Codesniffer: Add sniff to detect spaces before comma in argument lists - https://phabricator.wikimedia.org/T168970#3382818 (10Legoktm) Is there ever a situation where there should be any whitespace before a comma? [17:48:42] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10WorkType-NewFunctionality: Configure phabricator clustering for daemons and repositories - https://phabricator.wikimedia.org/T143175#3383677 (10Volans) @mmodell thanks for the additional info. [17:55:47] (03Draft1) 10Paladox: Update slave-scripts/bin/php wrapper script to support php 7.0 [integration/jenkins] - 10https://gerrit.wikimedia.org/r/361700 [17:55:49] (03PS2) 10Paladox: Update slave-scripts/bin/php wrapper script to support php 7.0 [integration/jenkins] - 10https://gerrit.wikimedia.org/r/361700 [17:57:26] 17:55:47 [Tue Jun 27 17:55:46 2017] [hphp] [1025:7fa3ef6cb200:0:000002] [] Uncaught exception: Could not open extension /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory [17:57:40] 10Gerrit, 10MediaWiki-Extensions-Pickle, 10Repository-Admins: Archive Pickle repo at Gerrit - https://phabricator.wikimedia.org/T167610#3383734 (10jeblad) I believe this is archived now? [17:58:33] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Uncaught exception: Could not open extension when trying to load tidy.so - https://phabricator.wikimedia.org/T168978#3383754 (10Legoktm) [17:59:56] (03CR) 10jerkins-bot: [V: 04-1] Update slave-scripts/bin/php wrapper script to support php 7.0 [integration/jenkins] - 10https://gerrit.wikimedia.org/r/361700 (owner: 10Paladox) [18:01:16] It's only affecting hhvm. [18:02:14] i think this may https://gerrit.wikimedia.org/r/#/c/342635/5/modules/contint/manifests/packages/php.pp be it affecting [18:02:15] things [18:02:27] but not 100% sure as it is not in the style of php-* [18:02:33] legotkm ^^ [18:02:41] ? [18:02:41] legoktm ^^ [18:02:59] the packages are "tidy" [18:03:08] but not sure if that is what is causing the problem. [18:03:09] I am on it [18:03:43] thanks hashar :) [18:03:57] hashar: are you looking into the tidy.so thing? [18:04:30] 10Continuous-Integration-Infrastructure: CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004#3383785 (10hashar) [18:04:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004#3383797 (10hashar) p:05Triage>03Unbreak! a:03ha... [18:04:41] 10MediaWiki-Codesniffer: Add sniff to detect spaces before comma in argument lists - https://phabricator.wikimedia.org/T168970#3383801 (10Umherirrender) Not sure, but possible is: ``` list( , , $var ) ``` [18:05:15] that's a duplicate [18:05:20] legoktm: yes [18:05:20] https://phabricator.wikimedia.org/T169004 [18:05:27] matmarex poked about it eariler [18:05:34] but I was having dinner with kids .. [18:05:47] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Uncaught exception: Could not open extension when trying to load tidy.so - https://phabricator.wikimedia.org/T168978#3383807 (10Paladox) [18:05:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004#3383809 (10Paladox) [18:05:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004#3383810 (10Legoktm) [18:05:55] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Uncaught exception: Could not open extension when trying to load tidy.so - https://phabricator.wikimedia.org/T168978#3383811 (10Legoktm) [18:06:03] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Uncaught exception: Could not open extension when trying to load tidy.so - https://phabricator.wikimedia.org/T168978#3383813 (10Legoktm) a:03hashar [18:06:25] paladox linked https://gerrit.wikimedia.org/r/#/c/342635/ which looks pretty suspect [18:06:48] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3383814 (10demon) ``` gerrit> select * from accounts where full_name = 'MarcoAurelio'; registered_on | full_name | preferred_email... [18:06:57] somehow that broke it for me on a test slave. [18:07:16] broke it because i was using stretch, but the class worked up until that was merged. [18:08:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004#3383817 (10hashar) Yeah the `hhvm-tidy` package is n... [18:08:37] fuck it puppet [18:08:41] (03PS1) 10Jdlrobson: Remove Cards from release tools [tools/release] - 10https://gerrit.wikimedia.org/r/361703 (https://phabricator.wikimedia.org/T167452) [18:09:29] and deb packages [18:10:51] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3383829 (10Paladox) @demon thank you for doing that :) i will share your findings to upstream hopefully they will manage to fix it. [18:10:58] (03CR) 10jerkins-bot: [V: 04-1] Remove Cards from release tools [tools/release] - 10https://gerrit.wikimedia.org/r/361703 (https://phabricator.wikimedia.org/T167452) (owner: 10Jdlrobson) [18:12:56] so bascally [18:13:05] hhvm-tidy and libtidy-dev conflicts on jessie :((( [18:18:41] !log Regenerating Jessie nodepool image to hopefulyl bring back hhvm-tidy package - T169004 [18:18:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:18:44] T169004: CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004 [18:18:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004#3383877 (10hashar) 05duplicat... [18:19:25] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Uncaught exception: Could not open extension when trying to load tidy.so - https://phabricator.wikimedia.org/T168978#3383882 (10hashar) Sorry I havent seen that task and used another one instead: T... [18:19:36] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Uncaught exception: Could not open extension when trying to load tidy.so - https://phabricator.wikimedia.org/T168978#3383888 (10hashar) [18:19:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004#3383785 (10hashar) [18:21:29] ahh that is because of sury [18:21:59] sury? [18:22:02] php7? [18:23:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004#3383899 (10hashar) `hhvm-tidy`... [18:23:47] paladox: yes sury provides a bunch of packages [18:23:58] yeh [18:23:59] and libtidy-dev is one of them but it is m arked as breaking a dependency of hhvm-tidy [18:24:01] .. [18:24:28] 2017-06-27 18:24:05,661 INFO nodepool.image.build.wmflabs-eqiad.snapshot-ci-jessie: hhvm-tidy (0.1.3~jessie1 => 0.1.3~jessie2) [18:24:30] progress [18:24:46] we could mark it as not allowed to install [18:31:53] !log Image snapshot-ci-jessie-1498587497 in wmflabs-eqiad is ready - T169004 [18:31:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:31:57] T169004: CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004 [18:32:18] thanks hashar :) [18:38:19] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:26] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:34] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:48] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:56] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:11] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [18:39:15] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:39:42] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:48] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:03] PROBLEM - Puppet errors on deployment-memc04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:40:11] those are caused by broken puppet [18:40:13] see -cloud [18:40:15] PROBLEM - Puppet errors on deployment-ms-fe02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:40:28] PROBLEM - Puppet errors on deployment-prometheus01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:34] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:36] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:40:38] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:40:53] PROBLEM - Puppet errors on deployment-puppetdb01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:41:00] PROBLEM - Puppet errors on deployment-stream is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [18:41:10] PROBLEM - Puppet errors on deployment-kafka03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:21] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:42:26] 10MediaWiki-Codesniffer: Adjust MediaWiki.WhiteSpace.SpaceBeforeClassBrace.NoSpaceBeforeBrace for use with multiline class statements - https://phabricator.wikimedia.org/T168961#3383968 (10Umherirrender) This seems to be the reason why this sniff is disabled on mediawiki/core. You can find more examples in /inc... [18:42:45] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:42:47] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:42:48] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:48] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:52] PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:43:05] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:43:07] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:43:11] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:43:11] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:43:11] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:43:13] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [18:43:21] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:43:26] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:45:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Figure out the package conflict between libtidy-dev from sury and hhvm-tidy - https://phabricator.wikimedia.org/T169008#3383969 (10hashar) [18:45:12] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:45:16] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:45:30] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:47:10] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:47:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004#3383990 (10hashar) 05Open>0... [18:47:24] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:48:02] PROBLEM - Puppet errors on deployment-zookeeper02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:48:23] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:48:27] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:48:47] PROBLEM - Puppet errors on deployment-sca04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:48:57] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:49:33] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:49:51] PROBLEM - Puppet errors on deployment-salt02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:50:57] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:51:12] PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:52:19] PROBLEM - Puppet errors on deployment-poolcounter04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:52:41] PROBLEM - Puppet errors on deployment-ms-be-test01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:52:51] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:52:51] PROBLEM - Puppet errors on deployment-apertium02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:53:27] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:53:49] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:54:25] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:54:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:54:57] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:55:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: CI job fails with: /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212/tidy.so: cannot open shared object file: No such file or directory - https://phabricator.wikimedia.org/T169004#3384045 (10hashar) I have poke... [18:56:22] (03CR) 10Hashar: [C: 032] Update Wikidata to wmf/1.30.0-wmf.7 [tools/release] - 10https://gerrit.wikimedia.org/r/361666 (owner: 10Aude) [18:57:12] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:57:13] PROBLEM - Puppet errors on deployment-puppetmaster02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:57:23] (03Merged) 10jenkins-bot: Update Wikidata to wmf/1.30.0-wmf.7 [tools/release] - 10https://gerrit.wikimedia.org/r/361666 (owner: 10Aude) [19:00:17] legoktm: issue solved! [19:17:23] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:27:13] RECOVERY - Puppet errors on deployment-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:30:58] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:56] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:32:12] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:32:20] RECOVERY - Puppet errors on deployment-poolcounter04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:32:52] RECOVERY - Puppet errors on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:33:26] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:33:48] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:24] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:40] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:41] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:57] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [19:35:27] RECOVERY - Puppet errors on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:37:39] RECOVERY - Puppet errors on deployment-ms-be-test01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:37:47] RECOVERY - Puppet errors on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:37:49] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:09] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:19] RECOVERY - Puppet errors on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:33] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [19:38:50] RECOVERY - Puppet errors on deployment-trending01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:39:52] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [19:40:04] thcipriani: got a sec re scap on deployment-tin? [19:40:34] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:40:52] RECOVERY - Puppet errors on deployment-puppetdb01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:41:10] RECOVERY - Puppet errors on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:42:50] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:43:08] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:43:25] RECOVERY - Puppet errors on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:43:43] mobrovac: just ran to get lunch. I'll be back at my desk in 10-15min [19:43:59] kk no pb [19:44:03] buon appetito! [19:44:11] RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [19:45:39] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:45:59] RECOVERY - Puppet errors on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [19:47:19] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [19:48:03] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:48:10] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:48:14] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:48:22] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:49:16] RECOVERY - Puppet errors on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [19:49:18] the test queue looks backlogged - is that because instances are being rebuilt? [19:50:04] RECOVERY - Puppet errors on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:50:10] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:50:14] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:50:14] RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:50:32] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [19:50:34] RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:52:08] 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10MW-1.28-release (WMF-deploy-2016-10-25_(1.28.0-wmf.23)), 10Patch-For-Review, 10Wikimedia-log-errors: logspam from mediawiki - LoadBalancer::{closure}: found writes/callbacks pending. - https://phabricator.wikimedia.org/T149353#3384237 (... [19:52:11] RECOVERY - Puppet errors on deployment-ms-be03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:52:25] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:52:45] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:52:45] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:53:09] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [19:53:23] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:53:23] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:54:27] MaxSem: not sure why but it is very much backlogged [19:55:39] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:56:12] RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:57:52] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:58:04] RECOVERY - Puppet errors on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:58:26] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:58:48] RECOVERY - Puppet errors on deployment-sca04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:58:58] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:59:06] jenkins seems to be processing jobs, not sure where the backlog came from [19:59:34] RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:59:48] RECOVERY - Puppet errors on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:03:50] mobrovac: back. Scap on deployment-tin? [20:04:59] thcipriani: yeah, i cherry-picked https://gerrit.wikimedia.org/r/#/c/360686/ to get recommendation-api deploy repo in, and puppet is barking on deployment-tin now [20:05:09] thcipriani: mind running it there to see what it says? [20:05:27] * thcipriani does [20:08:19] hrm, I wonder why the git clone doesn't work...seems to create the directory [20:09:49] well. ran the same command puppet was complaining about and it worked manually... [20:09:52] * thcipriani fiddles [20:10:09] hm :/ [20:10:31] i konw that i used the approach myself in the past, but would be nice to finally figure this out :) [20:10:38] indeed [20:10:50] I feel like I've seen this happen before as well [20:15:36] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T167536#3384300 (10mmodell) p:05Normal>03High [20:15:46] yup yup. you did thcipriani :) [20:18:27] mobrovac: mind if I break this thing a few times? Only way to make an omelette... [20:19:01] sure thcipriani, go ahead and make that omelette :) [20:19:07] i'm in a meeting any way [20:19:11] kk [21:04:26] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10media-storage, 10Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3384428 (10hashar) 05Open>03Resolved a:03hashar Lo... [21:04:33] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T167536#3384434 (10mmodell) [21:11:48] 10Release-Engineering-Team (Kanban), 10Labs, 10Labs-Infrastructure: labnet-users group can no more access labnet1001 / labnet1002 - https://phabricator.wikimedia.org/T169018#3384480 (10hashar) [21:13:15] 10Release-Engineering-Team (Kanban), 10Labs, 10Labs-Infrastructure: labnet-users group can no more access labnet1001 / labnet1002 - https://phabricator.wikimedia.org/T169018#3384495 (10hashar) I had noticed the lack of access for a few weeks already. In puppet site.pp, both hosts have the role `labs::openst... [21:19:25] 10Release-Engineering-Team (Kanban), 10Labs, 10Labs-Infrastructure, 10Patch-For-Review: labnet-users group can no more access labnet1001 / labnet1002 - https://phabricator.wikimedia.org/T169018#3384508 (10hashar) p:05Triage>03Low Broken since April 10th, but apparently I am the sole user relying on it... [21:19:58] 10Release-Engineering-Team (Kanban), 10Labs, 10Labs-Infrastructure, 10Patch-For-Review: labnet-users group can no more access labnet1001 / labnet1002 - https://phabricator.wikimedia.org/T169018#3384480 (10hashar) a:03hashar [21:20:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Figure out the package conflict between libtidy-dev from sury and hhvm-tidy - https://phabricator.wikimedia.org/T169008#3384512 (10hashar) a:05hashar>03None [21:36:19] thcipriani: any luck? [21:36:52] no :( [21:37:36] thought I had it, it's weird when you do: sudo -u trebuchet -g wikidev -- blah as root it throws a permissions error, but I made a sudoers rule, now the command works manually [21:37:51] I am looking at puppet source code right now. That's when you know you're having a bad time [21:58:34] !log mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki wikidatawiki --new-data-type 'string' --property-id P34 [21:58:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:06:46] indeed thcipriani, i remember those painful moments :) [22:10:18] mobrovac: https://gist.github.com/thcipriani/1d7230e77878a8828f992ca227bbc7c8 slowly running out of ideas :) [22:11:35] which seems to be what puppet is doing https://github.com/puppetlabs/puppet/blob/master/lib/puppet/util/execution.rb#L268 [22:23:22] 10Deployment-Systems, 10MediaWiki-Internationalization, 10Performance-Team, 10Performance: Experiment with plain .php files for l10n cache instead of CDB - https://phabricator.wikimedia.org/T99740#3384597 (10Krinkle) >>! In T99740#1892621, @Reedy wrote: >>>! In T99740#1892455, @EBernhardson wrote: >> Since... [22:25:02] 10Deployment-Systems, 10MediaWiki-Internationalization, 10Performance-Team, 10Performance: Experiment with plain .php files for l10n cache instead of CDB - https://phabricator.wikimedia.org/T99740#1297948 (10demon) Yes, please. Let's. [22:38:56] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [22:42:43] !log cherry-picking gerrit:360891/2 [22:42:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:44:42] mobrovac: fixed! https://gerrit.wikimedia.org/r/#/c/361796/ [22:58:05] !log cherry-picking gerrit:360891/3 [22:58:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:25:17] wth: Error: 1146 Table 'jenkins_u0_mw.unittest_globaluser' doesn't exist (127.0.0.1:3306) [23:25:27] ^ this is from https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm-jessie/9184/console [23:35:39] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0]