[00:58:29] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:59:29] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:01:18] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:02:08] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:03:18] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [01:07:01] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:10:46] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:12:24] PROBLEM - Puppet run on deployment-mediawiki06 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:16:53] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:17:27] PROBLEM - Puppet run on deployment-mediawiki05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:18:09] PROBLEM - Puppet run on deployment-jobrunner02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:19:15] PROBLEM - Puppet run on deployment-zotero01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:19:22] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:19:34] PROBLEM - Puppet run on deployment-parsoid09 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:19:48] PROBLEM - Puppet run on deployment-sca03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:24:00] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:25:05] PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:25:19] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:27:13] PROBLEM - Puppet run on deployment-mediawiki04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:30:12] alright who broke it [01:31:17] twentyafterfour, "The following packages will be DOWNGRADED: scap" [01:31:26] thcipriani|afk [01:31:40] Krenair: :-/ [01:36:20] I think jenkins did it [01:36:22] :-/ [01:40:00] yeah I assumed that much [01:40:30] I ran a rebuild on the last known good version [01:51:51] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [01:54:36] RECOVERY - Puppet run on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [01:57:26] RECOVERY - Puppet run on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [01:58:08] RECOVERY - Puppet run on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:59:13] RECOVERY - Puppet run on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:59:25] RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [01:59:47] RECOVERY - Puppet run on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [02:02:13] RECOVERY - Puppet run on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [02:04:03] RECOVERY - Puppet run on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [02:05:04] RECOVERY - Puppet run on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:05:18] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [02:06:18] RECOVERY - Puppet run on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:07:08] RECOVERY - Puppet run on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:08:31] RECOVERY - Puppet run on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:09:27] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [02:12:00] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [02:13:20] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:20:45] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:22:39] RECOVERY - Puppet run on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [02:48:01] 10MediaWiki-Releasing, 10Timeless, 10Vector, 10Wikimedia-Developer-Summit (2017): Replacing Vector as the default MediaWiki skin - https://phabricator.wikimedia.org/T149636#2784643 (10Isarra) 05Open>03declined ...nevermind. [04:11:37] 10Browser-Tests-Infrastructure, 07Ruby, 15User-zeljkofilipin: Mediawiki Ruby gem incorrectly assumes path to index.php - https://phabricator.wikimedia.org/T149169#2784795 (10Oreolek) Here's my example code: https://git.oreolek.ru/oreolek/mediawiki_mwe [06:48:29] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:27:34] 03Scap3, 10ContentTranslation-CXserver, 10MediaWiki-extensions-ContentTranslation, 06Services, and 4 others: Enable Scap3 config deploys for CXServer - https://phabricator.wikimedia.org/T147634#2784896 (10KartikMistry) a:03KartikMistry [07:28:06] 03Scap3, 10ContentTranslation-CXserver, 10MediaWiki-extensions-ContentTranslation, 06Services, and 4 others: Enable Scap3 config deploys for CXServer - https://phabricator.wikimedia.org/T147634#2699130 (10KartikMistry) Ping @mobrovac for estimated deadline of this task. [07:28:30] RECOVERY - Puppet run on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [07:45:10] (03PS1) 10Hashar: [ParsoidBatchAPI] add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/320730 [07:48:49] (03CR) 10Hashar: [C: 032] [ParsoidBatchAPI] add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/320730 (owner: 10Hashar) [07:49:35] (03Merged) 10jenkins-bot: [ParsoidBatchAPI] add npm test [integration/config] - 10https://gerrit.wikimedia.org/r/320730 (owner: 10Hashar) [07:54:10] 10Continuous-Integration-Config, 07I18n, 13Patch-For-Review: Wikimedia deployed extensions lack npm entry point / banana checker - https://phabricator.wikimedia.org/T150332#2784904 (10hashar) [08:13:20] 03Scap3, 10ContentTranslation-CXserver, 10MediaWiki-extensions-ContentTranslation, 06Services, and 3 others: Enable Scap3 config deploys for CXServer - https://phabricator.wikimedia.org/T147634#2784929 (10Arrbee) [09:08:05] (03PS1) 10DCausse: Add SiteMatrix to CirrusSearch dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/320741 [09:11:08] 10Continuous-Integration-Config, 10FR-Smashpig, 10fundraising-tech-ops: Switch wikimedia/fundraising/SmashPig to use tox as an entry point - https://phabricator.wikimedia.org/T114361#2784964 (10hashar) [09:11:54] 10Continuous-Integration-Config, 10FR-Smashpig, 10fundraising-tech-ops: Switch wikimedia/fundraising/SmashPig to use tox as an entry point - https://phabricator.wikimedia.org/T114361#1693094 (10hashar) [09:14:03] 10Continuous-Integration-Config, 10FR-Smashpig, 10fundraising-tech-ops: Switch wikimedia/fundraising/SmashPig to use tox as an entry point - https://phabricator.wikimedia.org/T114361#2784994 (10hashar) Given there is only a few python scripts, I am not sure it is worth the hassle of adding a python linter to... [09:26:40] !log Regenerate Nodepool base image for Jessie and refreshing snapshot image [09:26:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:32:56] 10Continuous-Integration-Infrastructure, 06Operations: Ubuntu Trusty mirror has Packages Hash Sum mismatch errors - https://phabricator.wikimedia.org/T150406#2785045 (10hashar) [09:33:12] !log Image ci-jessie-wikimedia-1478770026 in wmflabs-eqiad is ready [09:33:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:39:22] 10Continuous-Integration-Infrastructure, 06Operations: Ubuntu Trusty mirror has Packages Hash Sum mismatch errors - https://phabricator.wikimedia.org/T150406#2785065 (10hashar) Looks like the checksum in the`InRelease` files does not match / the file is more recent. [09:39:31] 10Continuous-Integration-Infrastructure, 06Operations: Ubuntu Trusty mirror has Packages Hash Sum mismatch errors - https://phabricator.wikimedia.org/T150406#2785066 (10hashar) p:05Triage>03Low [09:40:57] 10Continuous-Integration-Config, 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#2785079 (10hashar) Ahh the coupling with MediaWiki is due / related to T107529 [09:44:34] hashar: are all builds failing due to betafeatures issues? [09:44:43] (sorry to target you directly) [09:47:32] hashar: sorry -- i'll raise a bug, i think the oojs-ui update broke betafeatures unit tests [09:51:53] oh man :( [09:52:12] phuedx: we have a task for oojs upgrade that broke the build yesterday [09:52:23] huzzah! [09:54:14] zeljkof: gotta postpone our chat session to fix up above^^ :D [09:54:26] phuedx: got a link to a failing build ? [09:55:02] hashar: sure, ping me when you are ready [09:55:36] (looks at Popups change https://gerrit.wikimedia.org/r/#/c/320606/ ) [09:57:22] oh my god [09:57:49] https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/28315/console some HTML diff fails [10:00:49] https://integration.wikimedia.org/ci/job/mwext-testextension-php55/26598/console [10:01:00] hashar: ^ [10:02:12] yeah [10:02:17] I think HTML is deprecated [10:02:24] you should use json like all the cool kids [10:02:30] hashar: lol [10:02:36] i can update the tests if you've got a chat lined up [10:02:43] and really move to React [10:02:52] going to try BetaFeatures [10:03:39] dummy patch for BetaFeatures on https://gerrit.wikimedia.org/r/320748 [10:03:57] and there is a recent change on BetaFeatures "Use OOUI checkboxes" https://gerrit.wikimedia.org/r/#/c/315850/ [10:05:00] so [10:05:06] BetaFeatures fails [10:05:14] (I am merely speaking to myself / brain dumping here) [10:05:29] https://integration.wikimedia.org/ci/job/mwext-testextension-php55/26623/console [10:05:33] shows it has no other dependencies [10:06:45] and I reproduce locally! [10:08:27] hashar: i've fixed the test cases [10:08:37] oh really [10:08:41] i'm just finding the change to ooui that introduced the /> to the checkbox [10:08:43] one sec [10:09:25] and I guess I will add BetaFeatures to the common job [10:09:31] (job I should really rename) [10:09:59] ahhh [10:10:57] mayn't have been oojs-ui actually [10:11:00] yeah the test does some regex matching [10:11:52] it is [10:12:14] I guess BetaFeatures relies on MediaWiki to generate some UI, which ultimately uses oojs-ui [10:12:29] and oojs-ui has been made to close the element with /> [10:12:56] hashar: https://github.com/wikimedia/oojs-ui/commit/cb96deb225c9bc930f3f39d5bce5f460c47bb7d1 [10:12:56] in MediaWiki if I rollback oojs from v0.18.0 v0.17.10 (with: git revert 291f2c0 ; composer install ) [10:13:04] the BetaFeatures test pass just fine [10:13:05] ^ that's the commit [10:13:09] \o/ [10:13:17] the test was expecting non-self-closing tags [10:13:22] so easy fix [10:13:26] just update the test case [10:13:28] i.e. rather than [10:13:29] yup [10:13:38] a better way would be to have the betafeatures test to compare at DOM level :] [10:13:40] just updating the commit message to include the ref that borked it [10:13:42] ^^^ [10:13:45] absolutely [10:13:50] but why bother? ;] [10:14:08] first fix the broken thing [10:14:16] second improve the broken thing [10:14:42] it is up to you guys to figure out whether it is worth the time investment [10:14:56] I am not sure how well one can do DOM parsing / comparison with PHP/PHPUnit [10:15:32] PROBLEM - Puppet run on deployment-pdfrender02 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [10:16:20] hashar: i agree that the tests are brittle [10:16:39] and that it's on //someone// to fix the test [10:17:15] oh [10:17:20] i'm just trying to find an avenue for that someone not to be me ;) [10:17:20] I am just being pedantic [10:17:26] the regex is probably good enough [10:17:37] I would just moneky patch the test case [10:17:41] get it merged [10:17:51] and forget / pretend the regex dom parsing is not an issue [10:17:59] then move to something else :D [10:18:43] phuedx: point me to the Gerrit change whenever you get it. Will be happy to review/+2 it [10:18:54] zeljkof: ok ready. Let me grab the webcam /headphones etc [10:20:26] hashar: https://gerrit.wikimedia.org/r/#/q/If950786aa130cb9dd39982065de3d8842919060e [10:20:31] RECOVERY - Puppet run on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:20:40] hashar: lol -- i like the way you do business! [10:20:52] well [10:21:14] when I am going to waste my time, I am always thinking at how much WMF donor money I am going to sink in a hole [10:21:17] and from there either: [10:21:25] A) I claim it is a pet project, does not invoice it [10:21:37] B) work over hours as a volunteer [10:21:46] C) abandon the crappy idea and move to something else [10:21:51] I often pick C :D [10:23:17] phuedx: so from there that should fix up your Popups patch [10:23:50] zeljkof: I am in the hangouts (I think [10:50:07] hashar: o/ anything against me creating a new instance in deployment-prep to test Redis? We are going to replace all the mc* hosts in production with new hw and I'd need to test the procedure [10:50:14] before blowing up something [10:50:30] PLEASE BLOW UP STUFF AS NEEDED !!!!!!!!! [10:50:32] :D [10:50:36] \o/ [10:53:07] elukey: more seriously, beta s a shared platform [10:53:17] and I guess it is a good place to test a migration / new version [10:55:20] elukey: and I guess !log here as needed [10:55:41] hashar: never played in beta, but I could be able to test the whole procedure [11:04:21] hashar: what is the name of the beta project in horizon? [11:04:44] 10Continuous-Integration-Config, 10Reading Web Trending service, 06Reading-Web-Backlog, 03Reading-Web-Sprint-85-💩, and 2 others: Add CI to trending-edits repo - https://phabricator.wikimedia.org/T149601#2785261 (10Jdlrobson) Any theories @mobrovac ? [11:05:16] wait I need coffee [11:05:38] I keep confusing with names [11:05:38] sigh [11:05:52] scratch what I've said [11:06:05] PEBCAK [11:07:42] elukey: deployment-prep [11:08:07] yes yes it took me 5 mins to realize it [11:08:23] sometimes I discover unfixable bugs in my brain [11:08:33] race conditions etc.. [11:08:46] coffee usually temporary covers them [11:12:37] ah [11:12:47] I offload a bunch of processing to other brains [11:12:51] via Phabricator :D [11:14:42] :D [11:15:43] more seriously, I see that the redis instances are handling both job queues and user sessions, meanwhile in prod they are separate (rdb* and mc* hosts). [11:15:54] so I wouldn't like to attempt a full migration :D [11:16:17] the idea is to try to avoid dropping user sessions while migrating [11:16:50] so we'll set the new redis mc host to slave of the current one, then switch nutcracker's config and turn the slave into master [12:48:25] elukey: memcached having both job queues and user sessions [12:48:30] that can be split as needed I guess [12:48:57] seems in prod the sessions are in Redis [12:49:03] no idea where they are stored on beta though [12:49:16] ah no got confused sorry [12:49:17] ignore me [12:51:12] (03CR) 10Hashar: "That does not define the dependency :) CirrusSearch is part of a set of extensions that are all tested together and run jobs 'mediawiki-" [integration/config] - 10https://gerrit.wikimedia.org/r/320741 (owner: 10DCausse) [12:52:05] doh.. ^ :) [12:53:06] (03Abandoned) 10DCausse: Add SiteMatrix to CirrusSearch dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/320741 (owner: 10DCausse) [12:53:16] dcausse: yeah the change you made is creative but not doing anything :] [12:53:22] :) [12:53:24] jjb configuration is really an arcane [12:53:42] what you did is pass a variable "dependency" containing "SiteMatrix" to the job that does the whitespaces linting [12:54:19] damn, I'm really a noob :) [12:54:28] you can tell the impact to the job configuration by looking at integration-jjb-config-diff result [12:54:42] it generates all the job with the parent commit [12:54:47] then again all jobs with the proposed patchset [12:54:55] then diff the resulting jobs config [12:55:01] we all have been noob at something at some point! [12:55:37] so I am just going to add SiteMatrix to the shared job [12:55:39] and hope all is fine [12:55:41] so I understand that SiteMatrix is already part of the deps used by all mw extension unit tests jobs? [12:55:45] ah ok [12:56:00] that would explain some errors I see [12:56:12] any suspicious build log to share? [12:56:17] I dont mind looking at it [12:56:40] https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/81753/console [12:56:51] the error: Exception: Global with key wgSiteMatrixSites doesn't exist and cant be stashed [12:57:03] yeah [12:57:06] can be triggered is SiteMatrix is not available [12:57:44] AssertionError: Project mediawiki/extensions/SiteMatrix pipeline test must have either composer-validate or a composer-* job [12:57:45] ahh [13:03:16] ppooooor SiteMatrix [13:05:41] what could I do to fix this (I'm not sure to understand what the problem is) ? [13:09:53] 10Continuous-Integration-Config, 10Reading Web Trending service, 06Reading-Web-Backlog, 03Reading-Web-Sprint-85-💩, and 2 others: Add CI to trending-edits repo - https://phabricator.wikimedia.org/T149601#2785501 (10mobrovac) 05Open>03Resolved I added the symlink, which fixed the issue. [13:10:11] dcausse: I am adding to SiteMatrix a job that runs "composer test" [13:10:25] unrelated to your issue [13:10:32] but that is a requirement [13:10:41] hashar: ok, thanks for looking into this! [13:11:41] (03PS1) 10Hashar: [SiteMatrix] add composer test [integration/config] - 10https://gerrit.wikimedia.org/r/320768 (https://phabricator.wikimedia.org/T150423) [13:13:55] (03Restored) 10Hashar: Add SiteMatrix to CirrusSearch dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/320741 (owner: 10DCausse) [13:14:03] (03PS2) 10Hashar: Add SiteMatrix to the shared job [integration/config] - 10https://gerrit.wikimedia.org/r/320741 (owner: 10DCausse) [13:14:09] (03CR) 10Hashar: [C: 032] [SiteMatrix] add composer test [integration/config] - 10https://gerrit.wikimedia.org/r/320768 (https://phabricator.wikimedia.org/T150423) (owner: 10Hashar) [13:14:51] (03Merged) 10jenkins-bot: [SiteMatrix] add composer test [integration/config] - 10https://gerrit.wikimedia.org/r/320768 (https://phabricator.wikimedia.org/T150423) (owner: 10Hashar) [13:14:52] side effect, SiteMatrix got php code sniffer [13:16:43] (03PS1) 10Tobias Gritschacher: Enable browsertests for TwoColConflict extension [integration/config] - 10https://gerrit.wikimedia.org/r/320770 (https://phabricator.wikimedia.org/T149719) [13:16:58] (03CR) 10Hashar: [C: 032] Add SiteMatrix to the shared job [integration/config] - 10https://gerrit.wikimedia.org/r/320741 (owner: 10DCausse) [13:17:12] (03CR) 10Hashar: "Tested against the last merged change of SiteMatrix https://gerrit.wikimedia.org/r/#/c/320309/" [integration/config] - 10https://gerrit.wikimedia.org/r/320741 (owner: 10DCausse) [13:17:41] (03Merged) 10jenkins-bot: Add SiteMatrix to the shared job [integration/config] - 10https://gerrit.wikimedia.org/r/320741 (owner: 10DCausse) [13:17:50] (03PS2) 10Tobias Gritschacher: Enable browsertests for TwoColConflict extension [integration/config] - 10https://gerrit.wikimedia.org/r/320770 (https://phabricator.wikimedia.org/T149719) [13:18:36] dcausse: i love your patch https://gerrit.wikimedia.org/r/#/c/320757/ [13:18:47] dcausse: it has more tests than actual code :D [13:18:56] :) [13:19:02] it has a bunch of issue for sure [13:19:05] https://gerrit.wikimedia.org/r/#/c/320757/1/tests/unit/resources/wmf/langlist [13:19:10] that one is highly suspicious [13:19:17] pretty sure you can the lang list from somewhere else [13:20:44] ah ok, I'll look into this, basically I followed the code path to discover all the data needed by SiteMatrix and InterwikiLookup [13:20:49] anyway phpunit runs at https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/81758/console [13:20:59] SiteMatrix is scary [13:21:02] yes :/ [13:21:08] it is heavily coupled with the wmf config in operations/mediawiki-config [13:21:20] but I guess you can come up with your lists for some indepent unit testing [13:21:34] or super challenging: add operations/mediawiki-config in the loop :D [13:21:40] :) [13:21:44] that is where it goes scary [13:22:02] yes I'm not super happy to have to copy/munge some wmf-config data into the code [13:22:07] cause ideally all patch to mediawiki-config would have to run the CirrusSearch test your are introducing to make sure that does not break CirrusSearch [13:22:14] and we gotta support both wmf branch + master somehow [13:24:00] we added some tests to wmf-config, but same problem here, we needed to copy some code from core, e.g. SiteConf [13:24:02] (03CR) 10Hashar: [C: 032] Enable browsertests for TwoColConflict extension [integration/config] - 10https://gerrit.wikimedia.org/r/320770 (https://phabricator.wikimedia.org/T149719) (owner: 10Tobias Gritschacher) [13:24:12] dcausse: there is a task for it iirc [13:24:20] filled by Erik B. [13:24:26] he needed that for the avro schema validation [13:24:35] yes [13:24:49] then when sending a patch to operations/mediawiki-config [13:24:53] (03Merged) 10jenkins-bot: Enable browsertests for TwoColConflict extension [integration/config] - 10https://gerrit.wikimedia.org/r/320770 (https://phabricator.wikimedia.org/T149719) (owner: 10Tobias Gritschacher) [13:25:00] we would need to clone mediawiki/core + probably all extensions [13:25:05] going to be quite slow [13:25:14] (or maybe restrict to just the subset of ext we need) [13:25:18] and do that for both wmf branches [13:25:29] yes I understand, this is not optimal [13:25:50] (03CR) 10Hashar: "Deployed :]" [integration/config] - 10https://gerrit.wikimedia.org/r/320770 (https://phabricator.wikimedia.org/T149719) (owner: 10Tobias Gritschacher) [13:26:07] 10Continuous-Integration-Infrastructure, 06Operations: Ubuntu Trusty mirror has Packages Hash Sum mismatch errors - https://phabricator.wikimedia.org/T150406#2785538 (10faidon) 05Open>03declined We use Ubuntu's recommended mirroring method (a dual-pass rsync) to mirror from an official Ubuntu source, rsync... [13:27:07] hashar: and thanks for your help! [13:27:22] hmm [13:27:26] FIXED !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [13:27:36] \o/ [13:27:38] the pity [13:27:44] is that it takes half an hour to set that up [13:28:02] when I am pretty sure that a couple decade from now an AI would do all of that automagically [13:28:10] (and I will just retire to a beach) [13:28:38] nah you would have to maintain that AI stuff :) [13:28:45] oh [13:29:10] well really. I would have loved to work in AI and Graph [13:29:14] the math behind are all awesome [13:39:14] 10Continuous-Integration-Infrastructure, 06Operations: Ubuntu Trusty mirror has Packages Hash Sum mismatch errors - https://phabricator.wikimedia.org/T150406#2785549 (10hashar) The blog link is a fascinating read for the history geek I am. Thank you I heave learned a few things about the repositories layout.... [13:41:07] dcausse: cloning mw in mediawiki-config tests is https://phabricator.wikimedia.org/T115713 [13:43:05] hashar: yes that's it, thanks [13:43:23] dcausse: mind exposing your use case on the task? [13:43:30] sure [13:43:33] danke [13:43:38] and I guess we can resurect the task [13:43:45] and figure out a good solution to the problem [13:46:38] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #208: 04FAILURE in 2 min 37 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/208/ [13:48:18] actually I think how I can partially sovle it [13:53:23] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests, 07Technical-Debt: Clone mediawiki into mediawiki-config when running test's via jenkins - https://phabricator.wikimedia.org/T115713#2785578 (10dcausse) I have a somewhat similar use case but very different in the sense that I'd like to have mw-config a... [13:57:46] some patches seem to be failing currently due to issue in wikibase [14:11:00] Nikerabbit: patches? [14:11:30] PROBLEM - Puppet run on deployment-pdfrender02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:14:07] hashar: https://gerrit.wikimedia.org/r/#/c/320571/ [14:17:16] 00:05:15.483 1) Wikibase\Test\Repo\Api\ApiXmlFormatTest::testSetReferenceXmlFormat [14:17:16] 00:05:15.484 UsageException: This page has been protected to prevent editing or other actions. [14:17:17] bah [14:18:09] the Wikidata builds are at https://gerrit.wikimedia.org/r/#/q/project:mediawiki/extensions/Wikidata [14:21:40] Nikerabbit: Nemo_bis: no idea why that Translate change could trigger the error in Wikibase [14:21:55] Nor what that test is about [14:41:31] RECOVERY - Puppet run on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:20:13] zeljkof: if you have a minute can you try to generate the Gemfile.lock for https://gerrit.wikimedia.org/r/#/c/320613/? Job is failing with message Gemfile.lock corrupt and I'm wondering why.. [15:20:29] Tobi_WMDE_SW: let me see... [15:21:07] maybe some issue generating it on Windows vs Unix [15:26:12] Tobi_WMDE_SW: could be, I have generated the file on my machine, let's see if it works now [15:26:21] and please check if it works on your machine [15:27:42] zeljkof: great! seems it works.. will check it [15:34:31] hashar: it would not be the first time they make each other fail though, iirc [15:51:16] Yippee, build fixed! [15:51:17] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #223: 09FIXED in 29 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/223/ [16:09:48] dcausse: I managed to hack the Jenkins job to clone mediawiki/core wmf branches :D [16:13:49] (03PS1) 10Hashar: (WIP) clone wmf branches for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/320792 (https://phabricator.wikimedia.org/T115713) [16:13:51] hashar: really? [16:14:19] hashar: that would be very nice [16:14:27] 03Scap3, 15User-mobrovac: Smart-merge checks for different environments - https://phabricator.wikimedia.org/T149668#2786093 (10thcipriani) [16:21:26] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-Unit-tests, 13Patch-For-Review, 07Technical-Debt: Clone mediawiki into mediawiki-config when running test's via jenkins - https://phabricator.wikimedia.org/T115713#2786126 (10hashar) context ====== That came back up. The challenge... [16:21:37] dcausse: I have replied on the task [16:21:42] but in the end yeah that was not so hard [16:21:53] just clone mediawiki/core @ whatever is listed in wikiversions.json [16:22:01] ok :) [16:22:06] will needs a lot of polish up though [16:22:18] sure [16:22:24] namely [16:22:36] how can we get extensions to register tests that are solely for operations/mediawiki-config [16:22:48] I would call that some thing like "wikimedia-config-integration-test" amybe :D [16:22:54] we will find ou [16:22:55] t [16:23:18] 03Scap3, 15User-mobrovac: Smart-merge checks for different environments - https://phabricator.wikimedia.org/T149668#2786130 (10thcipriani) [16:23:20] let me know if you need some testing, we have a test in mw-config that uses a duplicated class copied from core [16:23:29] ah nice [16:23:41] if you can propose a new patch that clear out the duplicated class [16:23:50] we can then adjust it and try to use the mw core checkouts [16:23:56] ok will do [16:24:27] and the Jenkins job is deployed at https://integration.wikimedia.org/ci/job/operations-mw-config-phpunit-2/build [16:24:41] that ask a bunch of parameter, they can be filled to fetch whatever patch you manage to craft [16:24:58] ah ok, nice to test [16:27:19] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-Unit-tests, 13Patch-For-Review, 07Technical-Debt: Clone mediawiki into mediawiki-config when running test's via jenkins - https://phabricator.wikimedia.org/T115713#2786132 (10hashar) If one want to build the job with a operations/me... [16:27:28] dcausse: I am going out for the week-end [16:27:34] but we can catch up on that anytime next week [16:27:44] I guess Erik will comment on the task over night :] [16:27:45] hashar: sure, I'll give it tomorrow I think [16:27:53] thanks! :) [16:28:06] I dont think I will check anything tomorrow. Got folks at home [16:28:15] sure np [16:28:18] at least there is some baby step progress already [16:28:30] yes it's a good start [16:28:55] * hashar escapes [16:29:48] o/ [16:48:28] Hey everyone, my name is Nick and I currently work as an SDET, I've 6years experience in all between this, development and support work. I wanted to start working on an open source project to help sharpen my skills and get exposed to different setups. I'd like to be involved in tool creation or I have vast experience in automated testing. I'm currently digging through the wiki, does anyone require someone of my skillset? [17:15:56] PROBLEM - Puppet run on repository is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:50:55] RECOVERY - Puppet run on repository is OK: OK: Less than 1.00% above the threshold [0.0] [18:24:21] PROBLEM - Host integration-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.16.42) [18:42:45] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:42:57] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T150465#2786724 (10greg) [18:43:11] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T149927#2769078 (10greg) [18:53:06] 06Release-Engineering-Team, 10ArchCom-RfC, 06Developer-Relations, 06WMF-Legal, and 4 others: Create formal process for CREDITS files - https://phabricator.wikimedia.org/T139300#2786795 (10Legoktm) [18:53:19] 06Release-Engineering-Team, 10ArchCom-RfC, 06Developer-Relations, 06WMF-Legal, and 5 others: Create formal process for CREDITS files - https://phabricator.wikimedia.org/T139300#2786797 (10Legoktm) a:03bd808 [19:17:48] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:19:28] have a patch thats failing in jenkins, but not on vagrant. I know i can ssh into the integration slaves ... but is there anyways to actually re-run the tests from the command line (with hhvm -m debug for breakpoints and such) to actually see whats happening? [19:19:58] a quick look seems like we don't use unique workspaces, instead we setup and teardown the workspace for each test [19:23:02] yeah, workspaces are re-used for each test. Mostly not cleaned up. Your workspace is likely still on the jenkins node. [19:25:05] well, reused if they're run on a permanent node, non-nodepool [19:25:25] could mark node as offline for a minute to fiddle with the instance, but we've reduced the number of permanent nodes recently, so that might back things up a bit :\ [19:25:31] thcipriani: hmm, ok [19:26:15] thcipriani: i checked the machine (integration-slave-trusty-1003 a minute after it failed, but the reported workspace was empty /srv/jenkins-workspace/workspace/mediawiki-extensions-hhvm [19:26:39] there's something there now, but i see jenkins has started up a new test [19:27:42] maybe though i can just run the appropriate scripts with workspace set to my home dir ... might be worth trying i suppose [19:41:02] 06Release-Engineering-Team, 10ArchCom-RfC, 06Developer-Relations, 06WMF-Legal, and 6 others: Create formal process for CREDITS files - https://phabricator.wikimedia.org/T139300#2787100 (10demon) Removing 1.28 tag because it doesn't block a release. [20:06:49] 10Beta-Cluster-Infrastructure, 10Analytics: Set up a fake Pageview API endpoint for the beta cluster - https://phabricator.wikimedia.org/T150483#2787190 (10Tgr) [20:08:22] 10scap, 06Operations, 06Performance-Team, 07Epic: During deployment old servers may populate new cache URIs - https://phabricator.wikimedia.org/T47877#2787205 (10Krinkle) 05Open>03Resolved a:03Krinkle [20:11:06] 10Beta-Cluster-Infrastructure, 10Analytics, 10PageViewInfo: Deploy WikimediaPageViewInfo extension to beta cluster - https://phabricator.wikimedia.org/T129602#2787212 (10Tgr) 05Open>03Resolved a:03Tgr The extension has been deployed; I split the part about setting up a mock Pageview API endpoint to {T1... [20:12:00] 10Beta-Cluster-Infrastructure, 10Analytics: Set up a fake Pageview API endpoint for the beta cluster - https://phabricator.wikimedia.org/T150483#2787190 (10Tgr) >>! In T129602#2787212, @Tgr wrote: > [using live enwiki data in an extension on the beta cluster] is fine for testing the extension but not useful fo... [20:21:31] 06Release-Engineering-Team, 10ArchCom-RfC, 06Developer-Relations, 06WMF-Legal, and 6 others: Create formal process for CREDITS files - https://phabricator.wikimedia.org/T139300#2426179 (10Legoktm) Okay, but I'd still like to see these patches included in 1.28. [20:28:40] 06Release-Engineering-Team, 10ArchCom-RfC, 06Developer-Relations, 06WMF-Legal, and 6 others: Create formal process for CREDITS files - https://phabricator.wikimedia.org/T139300#2787276 (10demon) That's fine, go ahead and do the backports. I'm just saying it's not worth blocking the release if you don't get... [20:46:02] 03Scap3, 15User-mobrovac: Smart-merge checks for different environments - https://phabricator.wikimedia.org/T149668#2787318 (10thcipriani) 05Open>03Resolved [20:48:48] PROBLEM - Puppet run on deployment-fluorine02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:57:50] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10Tool-Labs-tools-Zppixbot, 07Jenkins: CI for #zppixbot - https://phabricator.wikimedia.org/T150487#2787353 (10Zppix) [21:02:44] Zppix: around ? :) [21:02:47] yes [21:02:50] lets do the CI thinggie [21:02:53] ok [21:03:21] since we have so many projects and each and others have different preference [21:03:23] CI is dumb [21:03:30] for PHP it would just: [21:03:38] git clone && git fetch && git checkout yourpatch [21:03:39] then [21:03:45] composer install && composer test [21:04:11] so you want to add a composer.json file that describe the test tools and command to runs [21:04:18] eg https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#PHP [21:04:31] though that one would have your PHP adhere to MediaWiki coding standard (but that is probably a good thing) [21:04:47] (I should update that example) [21:05:03] all it should need is basic .php im using it as a webserver atm [21:06:26] ah just the php -l ? [21:10:50] yess [21:10:55] @ hasharAway [21:11:16] done [21:11:17] (03PS1) 10Hashar: php5 linter for labs/tools/ZppixBot [integration/config] - 10https://gerrit.wikimedia.org/r/320853 (https://phabricator.wikimedia.org/T150487) [21:11:49] grrrit-wm: restart [21:11:51] re-connecting to gerrit [21:11:52] reconnected to gerrit [21:12:06] thanks hasharAway [21:12:16] https://integration.wikimedia.org/ci/job/labs-tools-ZppixBot-php55lint/1/console [21:12:32] it just lint files that got changed in the patch [21:13:32] ok [21:13:47] (03CR) 10Hashar: [C: 032] php5 linter for labs/tools/ZppixBot [integration/config] - 10https://gerrit.wikimedia.org/r/320853 (https://phabricator.wikimedia.org/T150487) (owner: 10Hashar) [21:16:11] (03Merged) 10jenkins-bot: php5 linter for labs/tools/ZppixBot [integration/config] - 10https://gerrit.wikimedia.org/r/320853 (https://phabricator.wikimedia.org/T150487) (owner: 10Hashar) [21:17:17] Zppix: should be good now. Tested on https://gerrit.wikimedia.org/r/#/c/317301/1 [21:17:46] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10Tool-Labs-tools-Zppixbot, 07Jenkins, 13Patch-For-Review: CI for #zppixbot - https://phabricator.wikimedia.org/T150487#2787435 (10hashar) 05Open>03Resolved a:03hashar Should be good now. Tested on https://gerrit.wikimedia.org/r/#/c/317301/1 [21:36:50] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T149338#2787474 (10mmodell) [21:39:10] 03Scap3: scap rollback behavior problems - https://phabricator.wikimedia.org/T150267#2787479 (10mmodell) maybe the in_progress flag should be a log instead of a lock file? [21:41:27] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T149338#2787480 (10mmodell) 05Open>03Resolved [21:42:51] grrrit-wm: nick [21:42:53] Nick is already grrrit-wm not changing the nick. [21:47:57] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016), 07Documentation: Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2787504 (10Rfarrand) Hello! We just had our meeting today. We agree... [21:49:32] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016), 07Documentation: Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2787508 (10Rfarrand) @Qgil Can you please help us find one person who... [21:56:26] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016), 07Documentation: Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2787524 (10ksmith) @Rfarrand : Great summary. Thanks! Perhaps IRC is... [22:42:52] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:12:36] twentyafterfour: Can anything bad happen if accidentally invoking scap sync or scap sync-file from a non-deployment host? E.g. mw1017? [23:12:41] I hope not. [23:12:51] I just did from mw1017 when intending to run scap pull. [23:12:56] aborted immediately after [23:13:19] Krinkle: I don't think so [23:13:25] twentyafterfour: I looks pretty scary [23:13:39] Krinkle i dont think so what wer're you trying to scap? [23:14:06] Zppix: Nothing, just making sure the canary server is in a clean state before I run some tests [23:14:30] Krinkle ah, i see no harm if anything all you did was clean up the harddrive a bit xD [23:14:31] hrm, scap sync will start with validating all php files in the staging dir, which should fail because it doesn't exist [23:15:12] thcipriani: It doesn't fail hard enough https://gist.github.com/Krinkle/3325763126e78554396f9cf1718559fd [23:15:20] it spams the console but continuos effortlessly [23:19:48] hrm. Somehow this command: https://github.com/wikimedia/scap/blob/master/scap/tasks.py#L120-L126 is exiting 0 when you pass in /srv/mediawiki-staging/multiversion and /srv/mediawiki-staging/wmf-config [23:20:33] when that succeeds it just rsyncs from the deployment host, so like scap pull [23:20:43] shouldn't be harmful, should figure out why it doesn't explode [23:22:52] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [23:27:07] thcipriani: Yeah, I was suprised with it pulling from deployment host instead of pushing to that, or instead of pushing to the rsync masters [23:27:14] which is good I guess, but somewhat unexpected. [23:28:13] https://github.com/wikimedia/scap/blob/8eadbca15769596fc0d0f4a256c0af2291c60870/scap/main.py#L348-L358 [23:28:26] I suppose it's fine for valid_syntax to pass if none of the files exist [23:29:06] thcipriani: I was just looking at that find command the other day. We should probably replace that whole pipeline of find | xargs | php with a python iterator [23:30:00] eh, I guess, I'd be more comfortable about it if it exploded. You never mean to be executing that when there's not an /srv/mediawiki-staging [23:30:18] indeed [23:30:31] we have the concept of a host context and a target context in scap3. Does some initial sanity stuff. [23:31:10] there's also the assert_user thing for some commands [23:31:34] thcipriani couldnt you guys just restore the prev version before the scap accident [23:32:31] Zppix: no harm no foul, afaik. Nothing needs to be restored as nothing broke. [23:32:58] had execution not been stopped via Ctrl-C, unclear what might have happened. [23:33:23] now we just need to figure out how to make that a little harder :)