[00:46:15] 10Deployment-Systems, 6WMF-Legal, 7Documentation, 7Software-Licensing: mediawiki/tools/scap is lacking a license - https://phabricator.wikimedia.org/T94239#1652084 (10Slaporte) >>! In T94239#1158916, @bd808 wrote: > It's a derivative of unlicensed code that was lifted from operations/puppet :( > > I'd be... [00:46:38] Yippee, build fixed! [00:46:38] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #2598: FIXED in 37 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/2598/ [01:40:52] 10Deployment-Systems, 5Patch-For-Review: foreachwikiindblist to run scripts on all wikis matching a dblist expression - https://phabricator.wikimedia.org/T101213#1652185 (10Krenair) 5Open>3Resolved Syntax: `foreachwikiindblist "all.dblist - excludethis.dblist" script.php` [04:03:40] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseView, 10Wikidata: Relative path reference to an image in core causing all CI tests failing since 2015-09-17 - https://phabricator.wikimedia.org/T113017#1652298 (10santhosh) 3NEW [04:04:01] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseView, 10Wikidata: Relative path reference to an image in core causing all CI tests failing since 2015-09-17 - https://phabricator.wikimedia.org/T113017#1652309 (10santhosh) [04:04:06] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-ContentTranslation, 5Patch-For-Review: ContentTranslation builds are failing with seemingly unrelated errors - 2015-09-17 - https://phabricator.wikimedia.org/T112968#1651244 (10santhosh) [04:30:35] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-ContentTranslation, 5Patch-For-Review: ContentTranslation builds are failing with seemingly unrelated errors - 2015-09-17 - https://phabricator.wikimedia.org/T112968#1652350 (10santhosh) a:3santhosh [04:30:53] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-ContentTranslation, 3LE-CX6-Sprint 4, 5Patch-For-Review: ContentTranslation builds are failing with seemingly unrelated errors - 2015-09-17 - https://phabricator.wikimedia.org/T112968#1651244 (10santhosh) [05:17:17] PROBLEM - Puppet failure on deployment-cache-parsoid04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:17:17] PROBLEM - Puppet failure on deployment-cache-mobile04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:23:10] 10Deployment-Systems, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 7I18n: is shown instead of the actual message on Special:ContentTranslation in production - https://phabricator.wikimedia.org/T112964#1652394 (10Amire80) p:5Unb... [05:25:14] 10Deployment-Systems, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 7I18n: is shown instead of the actual message on Special:ContentTranslation in production - https://phabricator.wikimedia.org/T112964#1652403 (10santhosh) I see... [06:36:25] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [07:55:22] 10Beta-Cluster, 15User-greg: Add AWight and EWulczyn to the deployment-prep Nova project - https://phabricator.wikimedia.org/T112927#1652577 (10hashar) I granted both you sudo rights on all machines. That is done on https://wikitech.wikimedia.org/wiki/Special:NovaSudoer by adding people to the sudo group `und... [07:59:22] 10Deployment-Systems, 6WMF-Legal, 7Documentation, 7Software-Licensing: mediawiki/tools/scap is lacking a license - https://phabricator.wikimedia.org/T94239#1652581 (10hashar) >>! In T94239#1652084, @Slaporte wrote: > Did anyone else contribute to this in operations/puppet? Potentially a lot of people. `sc... [08:26:08] 10Deployment-Systems, 6WMF-Legal, 7Documentation, 7Software-Licensing: mediawiki/tools/scap is lacking a license - https://phabricator.wikimedia.org/T94239#1652610 (10hashar) Historically the scripts were solely on the cluster under /home/wikipedia/bin and /home/wikipedia/sbin. Potential authors would be a... [08:28:00] I was wondering why this channel was a little too quiet: https://gerrit.wikimedia.org/r/239328 [08:35:13] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseView, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-09-15: Relative path reference to an image in core causing all CI tests failing since 2015-09-17 - https://phabricator.wikimedia.org/T113017#1652620 (10Bene) a:3Bene [09:30:40] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #725: FAILURE in 1 hr 20 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/725/ [10:21:22] (03PS1) 10Hashar: gitmailmap: generate a mailmap from USERINFO [tools/release] - 10https://gerrit.wikimedia.org/r/239343 [10:22:31] (03CR) 10Hashar: "Ie for patches on core such as:" [tools/release] - 10https://gerrit.wikimedia.org/r/239343 (owner: 10Hashar) [10:51:44] (03CR) 10Hashar: "Patch for mediawiki/core https://gerrit.wikimedia.org/r/#/c/239346/" [tools/release] - 10https://gerrit.wikimedia.org/r/239343 (owner: 10Hashar) [11:08:50] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseView, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-09-15: Relative path reference to an image in core causing all CI tests failing since 2015-09-17 - https://phabricator.wikimedia.org/T113017#1652799 (10Amire80) The CX merges... [12:19:07] (03PS1) 10Hashar: nodepool: set jenkins shell to bash [integration/config] - 10https://gerrit.wikimedia.org/r/239359 [12:27:03] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: colordiff: command not found - https://phabricator.wikimedia.org/T112821#1652885 (10hashar) [12:27:05] 5Continuous-Integration-Scaling: Switch a job to Nodepool instances - https://phabricator.wikimedia.org/T112750#1644858 (10hashar) [12:28:33] 5Continuous-Integration-Scaling, 6Labs: Nodepool can't SSH to spawn instances due to wmflabs provided instances due ssh algorithms - https://phabricator.wikimedia.org/T102459#1652887 (10hashar) 5Open>3declined a:3hashar We are using our own images with DIB. They are based on stock Jessie and hence do not... [12:37:08] 10Continuous-Integration-Config, 10Wikidata: [Task] run one of the Wikibase CI jobs with a different language - https://phabricator.wikimedia.org/T113037#1652916 (10JanZerebecki) 3NEW [12:38:03] 10Continuous-Integration-Config, 10Wikidata: [Task] run one of the Wikibase CI jobs with a different language - https://phabricator.wikimedia.org/T113037#1652925 (10JanZerebecki) p:5Triage>3Normal [12:41:14] 10Continuous-Integration-Config, 10Wikidata: [Task] run some Wikibase CI jobs with different languages - https://phabricator.wikimedia.org/T113037#1652929 (10JanZerebecki) [12:51:36] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseView, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-09-15: [Bug] Relative path reference to an image in core causing all CI tests failing since 2015-09-17 - https://phabricator.wikimedia.org/T113017#1652969 (10JanZerebecki) 5... [12:51:41] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-ContentTranslation, 3LE-CX6-Sprint 4, 5Patch-For-Review, 5WMF-deploy-2015-09-22_(1.26wmf24): ContentTranslation builds are failing with seemingly unrelated errors - 2015-09-17 - https://phabricator.wikimedia.org/T112968#1652971 (10JanZerebeck... [12:52:45] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-ContentTranslation, 3LE-CX6-Sprint 4, 5Patch-For-Review, 5WMF-deploy-2015-09-22_(1.26wmf24): ContentTranslation builds are failing with seemingly unrelated errors - 2015-09-17 - https://phabricator.wikimedia.org/T112968#1652977 (10Amire80) ... [12:55:54] (03PS3) 10Hashar: Make sure colordiff is installed on nodepool instances [integration/config] - 10https://gerrit.wikimedia.org/r/239047 (https://phabricator.wikimedia.org/T112821) (owner: 10Zfilipin) [12:55:57] (03PS1) 10Hashar: nodepool: run integration/config puppet manifest on setup [integration/config] - 10https://gerrit.wikimedia.org/r/239365 (https://phabricator.wikimedia.org/T112821) [13:13:48] 10Continuous-Integration-Config, 10Wiki-Loves-Monuments-Database: Run unit tests with Jenkins in labs/tools/heritage - https://phabricator.wikimedia.org/T113039#1653038 (10JeanFred) 3NEW a:3JeanFred [13:59:56] (03PS1) 10Jean-Frédéric: Add tox testing for labs/tools/heritage [integration/config] - 10https://gerrit.wikimedia.org/r/239375 (https://phabricator.wikimedia.org/T113039) [14:10:37] 10Browser-Tests, 10QuickSurveys, 5Patch-For-Review, 3Reading-Web-Sprint-56-Four Lions: QA: Setup browser tests on beta cluster so we can share test articles - https://phabricator.wikimedia.org/T112204#1628865 (10phuedx) [239158](https://gerrit.wikimedia.org/r/239158) has a -1 from @Legoktm. [15:30:33] (03PS1) 10Jforrester: Switch wikihiero over to npm [integration/config] - 10https://gerrit.wikimedia.org/r/239395 [15:42:54] 10Browser-Tests, 10QuickSurveys, 5Patch-For-Review, 3Reading-Web-Sprint-56-Four Lions: QA: Setup browser tests on beta cluster so we can share test articles - https://phabricator.wikimedia.org/T112204#1653488 (10Jdlrobson) @Legoktm can you reply to my comment? I'm not sure what I'm supposed to do here and... [15:43:18] 10Browser-Tests, 10QuickSurveys, 5Patch-For-Review, 3Reading-Web-Sprint-56-Four Lions: QA: Setup browser tests on beta cluster so we can share test articles - https://phabricator.wikimedia.org/T112204#1653489 (10Jdlrobson) p:5Normal>3High [15:43:57] 10Browser-Tests, 10QuickSurveys, 5Patch-For-Review, 3Reading-Web-Sprint-56-Four Lions: QA: Setup browser tests on beta cluster so we can share test articles - https://phabricator.wikimedia.org/T112204#1628865 (10Jdlrobson) (Upping priority given we are sending spamming emails due to failures of browser tes... [15:57:11] (03CR) 10Esanders: [C: 031] Switch wikihiero over to npm [integration/config] - 10https://gerrit.wikimedia.org/r/239395 (owner: 10Jforrester) [16:00:20] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata: [Task] add CI to ext:ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1653521 (10JanZerebecki) 3NEW [16:29:01] (03PS2) 10JanZerebecki: Correct a few outdated comment regarding dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/238180 [16:30:14] (03CR) 10JanZerebecki: [C: 032] Correct a few outdated comment regarding dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/238180 (owner: 10JanZerebecki) [16:33:03] (03Merged) 10jenkins-bot: Correct a few outdated comment regarding dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/238180 (owner: 10JanZerebecki) [17:00:42] (03PS6) 1020after4: A context manager for managing nested loggers [tools/scap] - 10https://gerrit.wikimedia.org/r/239028 [17:00:57] (03CR) 10jenkins-bot: [V: 04-1] A context manager for managing nested loggers [tools/scap] - 10https://gerrit.wikimedia.org/r/239028 (owner: 1020after4) [17:09:59] (03PS2) 10Krinkle: Switch wikihiero over to npm [integration/config] - 10https://gerrit.wikimedia.org/r/239395 (owner: 10Jforrester) [17:10:05] (03CR) 10Krinkle: [C: 032] Switch wikihiero over to npm [integration/config] - 10https://gerrit.wikimedia.org/r/239395 (owner: 10Jforrester) [17:10:37] (03PS2) 10Hashar: nodepool: run integration/config puppet manifest on setup [integration/config] - 10https://gerrit.wikimedia.org/r/239365 (https://phabricator.wikimedia.org/T112821) [17:10:39] (03PS4) 10Hashar: nodepool: contint::packages::base in images [integration/config] - 10https://gerrit.wikimedia.org/r/239047 (https://phabricator.wikimedia.org/T112821) (owner: 10Zfilipin) [17:10:41] (03PS1) 10Hashar: nodepool: properly update git cache [integration/config] - 10https://gerrit.wikimedia.org/r/239417 [17:10:57] (03Merged) 10jenkins-bot: Switch wikihiero over to npm [integration/config] - 10https://gerrit.wikimedia.org/r/239395 (owner: 10Jforrester) [17:13:17] 10Beta-Cluster: Design of all wikis at the beta-Cluster broken - https://phabricator.wikimedia.org/T113060#1653766 (10Luke081515) 3NEW [17:13:28] 10Beta-Cluster: Design of all wikis at the beta-Cluster broken - https://phabricator.wikimedia.org/T113060#1653776 (10Luke081515) p:5Triage>3Unbreak! [17:14:18] I think there is something to do... [17:14:38] but I looks awful, for example: http://deployment.wikimedia.beta.wmflabs.org/wiki/Special:AbuseLog [17:20:03] Project beta-update-databases-eqiad build #3021: FAILURE in 2.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/3021/ [17:22:50] 10Beta-Cluster: Design of all wikis at the beta-Cluster broken - https://phabricator.wikimedia.org/T113060#1653819 (10greg) ``` 21 < ori> greg-g: probably the less.php library change 17:21 < ori> let me see 17:21 < greg-g> huge uptick in fatals: https://logstash-beta.wmflabs.org/#/dashboard/elasti... [17:22:58] Luke081515: ^ [17:25:38] 10Beta-Cluster: Design of all wikis at the beta-Cluster broken - https://phabricator.wikimedia.org/T113060#1653831 (10greg) a:3ori [17:26:39] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/239395 [17:27:10] 10Beta-Cluster: Design of all wikis at the beta-Cluster broken - https://phabricator.wikimedia.org/T113060#1653850 (10ori) Yes, broken by I826adf9. Should be fixed by I43131887. [17:31:23] greg-g: Thanks for ping. Before reporting I don't know which kind of bug it could be, so I thing it's better to report fast then to have this error on prod later ;) [17:31:37] Luke081515: indeed, thank you :) [17:32:28] (03PS2) 10Hashar: nodepool: set jenkins shell to bash [integration/config] - 10https://gerrit.wikimedia.org/r/239359 [17:32:52] (03CR) 10Hashar: [C: 032] "Works:" [integration/config] - 10https://gerrit.wikimedia.org/r/239359 (owner: 10Hashar) [17:33:41] (03Merged) 10jenkins-bot: nodepool: set jenkins shell to bash [integration/config] - 10https://gerrit.wikimedia.org/r/239359 (owner: 10Hashar) [17:34:26] (03PS2) 10Hashar: nodepool: properly update git cache [integration/config] - 10https://gerrit.wikimedia.org/r/239417 [17:36:02] 10Beta-Cluster, 10Deployment-Systems: beta-update-databases-eqiad failed with exception in mwscript update.php - https://phabricator.wikimedia.org/T113065#1653919 (10greg) 3NEW [17:38:08] 10Beta-Cluster: Design of all wikis at the beta-Cluster broken - https://phabricator.wikimedia.org/T113060#1653937 (10ori) 5Open>3Resolved [17:39:13] (03CR) 10Hashar: [C: 032] "Because git pull in a mirror does not work :-D" [integration/config] - 10https://gerrit.wikimedia.org/r/239417 (owner: 10Hashar) [17:39:40] 10Beta-Cluster: Design of all wikis at the beta-Cluster broken - https://phabricator.wikimedia.org/T113060#1653941 (10Luke081515) Looks better now, thanks for that quick solution [17:40:31] 10Beta-Cluster, 10MediaWiki-General-or-Unknown: Design of all wikis at the beta-Cluster broken - https://phabricator.wikimedia.org/T113060#1653942 (10greg) [17:41:37] 10Beta-Cluster, 10Deployment-Systems: beta-update-databases-eqiad failed with exception in mwscript update.php - https://phabricator.wikimedia.org/T113065#1653947 (10greg) [17:45:13] i need to deploy a config change to the cluster, https://gerrit.wikimedia.org/r/#/c/238446/ . We are doing a rolling restart of the elasticsearch cluster to disable an insecure feature, and this config tells mediawiki to stop using the insecure feature [17:45:54] basically just a warning, unless someone objects [17:46:10] which cluster? [17:46:41] mediawiki. it changes the way one api call in the Translate extension issues it's query to elasticsearch [17:47:09] sorry, I can not help you with that review, I can just give +1 [17:48:38] i can deploy it, but its no-deploy friday so i wanted to give a heads up [17:51:40] oh, prod cluster? [17:52:06] ebernhardson: prod? [17:52:19] greg-g: yes [17:52:19] (03Merged) 10jenkins-bot: nodepool: properly update git cache [integration/config] - 10https://gerrit.wikimedia.org/r/239417 (owner: 10Hashar) [17:52:40] * greg-g nods [17:53:19] ebernhardson: need help? [17:55:21] chasemp: nah it synced out now. basically the service was guaranteed some downtime and we needed to change its config somewhere in the middle of the rolling-restart [17:55:36] option 2 was no downtime, but run the rolling restart twice. we decided not to do that :) [17:55:47] all good, I'm around in case and I'll keep helping supervise the restarts [18:09:55] 3Scap3: Make puppet trebuchet provider work with scap3 - https://phabricator.wikimedia.org/T113072#1654085 (10thcipriani) 3NEW a:3thcipriani [18:11:24] 10Deployment-Systems, 3Scap3: Make puppet trebuchet provider work with scap3 - https://phabricator.wikimedia.org/T113072#1654102 (10greg) [18:11:25] 10Deployment-Systems, 3Scap3, 5Patch-For-Review: Allow batch_size to be configured for each deploy stage - https://phabricator.wikimedia.org/T112841#1654103 (10greg) [18:11:27] 10Deployment-Systems, 3Scap3, 5Patch-For-Review: Scap3 should break up remote deploy tasks - https://phabricator.wikimedia.org/T109861#1654105 (10greg) [18:11:29] 10Deployment-Systems, 3Scap3, 7Documentation, 5Patch-For-Review: Add documentation of the new scap3 features to the scap docs - https://phabricator.wikimedia.org/T112554#1654104 (10greg) [18:12:04] 3Scap3: Canary deploy for scap3 - https://phabricator.wikimedia.org/T113073#1654106 (10thcipriani) 3NEW a:3thcipriani [18:17:31] (03PS1) 10Legoktm: Bump parsoidsvc-{repository}-roundtrip-test-check timeout to 10 [integration/config] - 10https://gerrit.wikimedia.org/r/239431 [18:17:59] (03CR) 10Subramanya Sastry: [C: 031] Bump parsoidsvc-{repository}-roundtrip-test-check timeout to 10 [integration/config] - 10https://gerrit.wikimedia.org/r/239431 (owner: 10Legoktm) [18:21:02] Yippee, build fixed! [18:21:02] Project beta-update-databases-eqiad build #3022: FIXED in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/3022/ [18:42:16] (03CR) 10Legoktm: [C: 032] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/239431 (owner: 10Legoktm) [18:51:17] (03Merged) 10jenkins-bot: Bump parsoidsvc-{repository}-roundtrip-test-check timeout to 10 [integration/config] - 10https://gerrit.wikimedia.org/r/239431 (owner: 10Legoktm) [19:06:31] (03PS2) 10Krinkle: Add tox testing for labs/tools/heritage [integration/config] - 10https://gerrit.wikimedia.org/r/239375 (https://phabricator.wikimedia.org/T113039) (owner: 10Jean-Frédéric) [19:06:42] (03CR) 10Krinkle: [C: 032] Add tox testing for labs/tools/heritage [integration/config] - 10https://gerrit.wikimedia.org/r/239375 (https://phabricator.wikimedia.org/T113039) (owner: 10Jean-Frédéric) [19:07:22] (03PS5) 10Hashar: nodepool: contint::packages::base in images [integration/config] - 10https://gerrit.wikimedia.org/r/239047 (https://phabricator.wikimedia.org/T112821) (owner: 10Zfilipin) [19:07:40] (03Merged) 10jenkins-bot: Add tox testing for labs/tools/heritage [integration/config] - 10https://gerrit.wikimedia.org/r/239375 (https://phabricator.wikimedia.org/T113039) (owner: 10Jean-Frédéric) [19:07:59] (03PS3) 10Hashar: nodepool: run integration/config puppet manifest on setup [integration/config] - 10https://gerrit.wikimedia.org/r/239365 (https://phabricator.wikimedia.org/T112821) [19:08:01] (03PS6) 10Hashar: nodepool: contint::packages::base in images [integration/config] - 10https://gerrit.wikimedia.org/r/239047 (https://phabricator.wikimedia.org/T112821) (owner: 10Zfilipin) [19:08:15] (03PS7) 10Hashar: nodepool: contint::packages::base in images [integration/config] - 10https://gerrit.wikimedia.org/r/239047 (https://phabricator.wikimedia.org/T112821) (owner: 10Zfilipin) [19:08:37] (03CR) 10Hashar: [C: 032] "Needs https://gerrit.wikimedia.org/r/#/c/239365/ to actually be applied in the snapshots." [integration/config] - 10https://gerrit.wikimedia.org/r/239047 (https://phabricator.wikimedia.org/T112821) (owner: 10Zfilipin) [19:08:44] (03PS4) 10Hashar: nodepool: run integration/config puppet manifest on setup [integration/config] - 10https://gerrit.wikimedia.org/r/239365 (https://phabricator.wikimedia.org/T112821) [19:08:52] (03CR) 10Hashar: [C: 032] nodepool: run integration/config puppet manifest on setup [integration/config] - 10https://gerrit.wikimedia.org/r/239365 (https://phabricator.wikimedia.org/T112821) (owner: 10Hashar) [19:09:30] (03Merged) 10jenkins-bot: nodepool: contint::packages::base in images [integration/config] - 10https://gerrit.wikimedia.org/r/239047 (https://phabricator.wikimedia.org/T112821) (owner: 10Zfilipin) [19:09:56] (03Merged) 10jenkins-bot: nodepool: run integration/config puppet manifest on setup [integration/config] - 10https://gerrit.wikimedia.org/r/239365 (https://phabricator.wikimedia.org/T112821) (owner: 10Hashar) [19:14:18] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling: Nodepool images need Gerrit replication for git-clone performance - https://phabricator.wikimedia.org/T87294#1654330 (10hashar) I could not use source-repositories properly and it does a non-bare clone. There is some code in https://ger... [19:15:41] 3Scap3: Support event logging propagation from target to deployment host - https://phabricator.wikimedia.org/T113085#1654343 (10dduvall) 3NEW [19:17:34] (03CR) 10Dduvall: "The separate deployment directory + symlink sounds like a sane approach. What needs to change in this patchset to support that? AFAICT, it" [tools/scap] - 10https://gerrit.wikimedia.org/r/238839 (https://phabricator.wikimedia.org/T109514) (owner: 10Dduvall) [19:19:00] 3Scap3: Support event logging propagation from target to deployment host - https://phabricator.wikimedia.org/T113085#1654351 (10dduvall) a:3dduvall [19:22:21] twentyafterfour: are you in the middle of fixing up your nested logger patch? i wanted to rebase the rollback patch on it and start the json serializer stuff from there [19:22:46] i can always get going now and rebase later [19:25:16] (03PS1) 10Hashar: nodepool: setup commands need root [integration/config] - 10https://gerrit.wikimedia.org/r/239443 [19:25:28] (03CR) 10Hashar: [C: 032] nodepool: setup commands need root [integration/config] - 10https://gerrit.wikimedia.org/r/239443 (owner: 10Hashar) [19:29:03] sorry I haven't reviewed/followed up on any of the scap patches [19:29:20] marxarelli: have you figured out the Sphinx /doc/ system ? [19:29:46] hashar: no worries [19:29:55] no, i haven't dug into sphinx yet [19:30:36] should be all about: tox -e doc && open docs/_build/html/index.html [19:30:45] and hack in docs/source/.... [19:33:03] (03Merged) 10jenkins-bot: nodepool: setup commands need root [integration/config] - 10https://gerrit.wikimedia.org/r/239443 (owner: 10Hashar) [19:48:30] PROBLEM - Puppet failure on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:50:17] 10Continuous-Integration-Config, 10ArticlePlaceholder, 10Wikidata: [Task] add CI to extension ArticlePlaceholder - https://phabricator.wikimedia.org/T113049#1654419 (10hashar) [19:56:16] (03PS1) 10Hashar: nodepool: run puppet on snapshot setup [integration/config] - 10https://gerrit.wikimedia.org/r/239448 [19:56:29] (03CR) 10Hashar: [C: 032] nodepool: run puppet on snapshot setup [integration/config] - 10https://gerrit.wikimedia.org/r/239448 (owner: 10Hashar) [19:57:14] (03Merged) 10jenkins-bot: nodepool: run puppet on snapshot setup [integration/config] - 10https://gerrit.wikimedia.org/r/239448 (owner: 10Hashar) [20:00:42] marxarelli: I am reworking it. If we were working on a branch we could just merge it and then I could start a new patch to fix it instead of reworking the existing patch [20:00:47] I really hate working in gerrit [20:00:54] (03CR) 10Jforrester: "Used in I84b7f1a4d9f." [integration/config] - 10https://gerrit.wikimedia.org/r/238985 (owner: 10Jforrester) [20:03:34] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/238974 (https://phabricator.wikimedia.org/T112204) (owner: 10Jdlrobson) [20:04:16] andrewbogott: so apparently Nodepool has booted 2000 instances already [20:04:27] I am not sure why though [20:04:44] hashar: and deleted 1990 of them I hope [20:05:20] yeah there is a setting to define the max # of instances for a provider [20:05:34] and we have the tenant quota as well [20:08:26] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/238974 (https://phabricator.wikimedia.org/T112204) (owner: 10Jdlrobson) [20:11:22] ./scap/tasks.py:80:38: E901 SyntaxError: invalid syntax [20:11:24] def check_valid_syntax(*paths, logger=None): [20:11:56] invalid syntax in the check_valid_syntax function? I see what you did there... [20:14:13] 5Continuous-Integration-Scaling: Switch a job to Nodepool instances - https://phabricator.wikimedia.org/T112750#1654523 (10hashar) [20:14:15] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: colordiff: command not found - https://phabricator.wikimedia.org/T112821#1654521 (10hashar) 5Open>3Resolved This is now solved ( https://integration.wikimedia.org/ci/job/integration-jjb-config-diff/3733/console )... [20:15:31] (03PS7) 1020after4: A context manager for managing nested loggers [tools/scap] - 10https://gerrit.wikimedia.org/r/239028 [20:15:56] (03CR) 10jenkins-bot: [V: 04-1] A context manager for managing nested loggers [tools/scap] - 10https://gerrit.wikimedia.org/r/239028 (owner: 1020after4) [20:16:14] marxarelli: ... I thought I fixed the errors, apparently jenkins disagrees [20:16:54] twentyafterfour: "ERROR: Unexpected indentation" [20:17:18] yeah ... [20:17:26] what the fuck, really python? [20:17:48] it doesn't want me to put code in a docstring? I guess I should use a regular comment? that's dumb [20:20:29] twentyafterfour: The clue was when you said 'python'. [20:22:15] what what ? [20:22:43] (03PS8) 1020after4: A context manager for managing nested loggers [tools/scap] - 10https://gerrit.wikimedia.org/r/239028 [20:23:11] oh that's just dumb [20:23:15] you can run it locally to catch it up faster [20:23:38] hashar: how do I run that locally? I got flake8 running but no idea about tox [20:23:55] got some spare cycles for a quick intro ? [20:24:24] python has virtualenv [20:24:25] yeah [20:24:35] yes [20:24:39] to create a somehow isolated version with some python ver [20:24:45] and bunch of modules [20:24:47] tox is a wrapper around virtualenv [20:24:53] oh... [20:24:59] and let you define your envs in a single file [20:25:20] AND it comes with predefined env for the python versions (ex: py27 , py34 ) [20:25:29] by default it installs your package using pip install . [20:25:30] so [20:25:36] if you tox -e py27 [20:26:03] do I use pip to install tox? [20:26:04] that creates a venv with python2.7, run setup.py install into it [20:26:11] and execute whatever command you gave him (ex: nosetests) [20:26:17] or sphinx [20:26:28] you can use your distro package but it is probably outdated [20:26:29] so yeah [20:26:45] if you have a mac / homebrew: pip install tox [20:26:51] else: pip install --user tox [20:27:01] I use linux but pip worked [20:27:01] that would create some /bin/tox somewhere in your homedir [20:27:08] tox -e py27 works [20:27:11] maybe ~/.local/bin/tox [20:27:15] then [20:27:28] running 'tox' executes all the env defined in tox.ini by envlist = [20:27:45] 'tox -l' list the envs that would be run this way [20:27:54] tox -e doc , run the 'doc' env, i.e. sphinx [20:28:15] doc: commands succeeded [20:28:17] congratulations :) [20:28:30] fun thing, you can pass parameters to tox and have them to be passed to the underlying command [20:28:48] so in theory one can: tox -e py27 -- whatever options for the command such as nosetests [20:29:51] cool, thanks hashar [20:29:55] seems to be working [20:30:31] (03CR) 10Thcipriani: "Would be a nice to have to just put the final finishing symlink in-place with a config var like `final_path` (ln -s [deploy_dir]/current " [tools/scap] - 10https://gerrit.wikimedia.org/r/238839 (https://phabricator.wikimedia.org/T109514) (owner: 10Dduvall) [20:31:52] (03PS1) 10Hashar: tox: support passing positional arguments to envs [tools/scap] - 10https://gerrit.wikimedia.org/r/239453 [20:31:59] twentyafterfour: ^^^ will let you test that one :} [20:43:04] (03CR) 1020after4: [C: 032] tox: support passing positional arguments to envs [tools/scap] - 10https://gerrit.wikimedia.org/r/239453 (owner: 10Hashar) [20:43:21] (03Merged) 10jenkins-bot: tox: support passing positional arguments to envs [tools/scap] - 10https://gerrit.wikimedia.org/r/239453 (owner: 10Hashar) [20:44:11] marxarelli, thcipriani, ostriches: I pushed what I think is current to the scap3 branch. May be rebase errors on my part. [20:44:38] I really needed some sort of integration branch though, we've got too many patches out there floating around in gerrit [20:45:40] twentyafterfour: awesome. i'll check it out [20:45:59] I also updated my gerrit patch for the logging context stuff [20:46:10] i think the best approach to rebase issues is just to keep patches small and get them reviewed/merged [20:46:22] if you based off of it, then it was just a few simple changes to fix flake8 errors [20:46:33] marxarelli: easier said than done [20:46:51] when it comes to refactoring it's hard to do it in bite sized pieces [20:47:16] especially without breaking scap at each step [20:47:44] very true [20:49:14] though we're shooting for backwards compatibility here, so it's good that scap breakage blocks merges [20:49:52] that's why there is a branch :) [20:52:50] I'm just trying to see the pieces all fit together [20:57:55] 10Deployment-Systems, 3Scap3: Canary deploy for scap3 - https://phabricator.wikimedia.org/T113073#1654741 (10greg) [20:58:28] 10Beta-Cluster, 10Deployment-Systems, 15User-greg: beta-update-databases-eqiad failed with exception in mwscript update.php - https://phabricator.wikimedia.org/T113065#1654744 (10greg) 5Open>3Resolved a:3greg Works again, dunno. [20:58:34] 10Beta-Cluster, 10Deployment-Systems, 15User-greg: beta-update-databases-eqiad failed with exception in mwscript update.php - https://phabricator.wikimedia.org/T113065#1654748 (10greg) a:5greg>3None [21:45:19] (03CR) 10Hashar: "https://gerrit.wikimedia.org/r/#/c/239446/ parent is unmerged, so gotta wait a bit :)" [integration/config] - 10https://gerrit.wikimedia.org/r/238985 (owner: 10Jforrester) [22:01:53] PROBLEM - Free space - all mounts on deployment-eventlogging02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlogging02.diskspace._srv.byte_percentfree (<100.00%) deployment-prep.deployment-eventlogging02.diskspace._var.byte_percentfree (<100.00%) [22:05:56] 10Deployment-Systems, 5Patch-For-Review: Adopt Semantic Versioning format for WMF deploy branches beginning with 1.27.0-wmf.1 - https://phabricator.wikimedia.org/T67306#1655031 (10greg) Just to be clear: I'm making the deployment calendar for the week of Sept 28th (ie, when we branch 1.27). It should name it... [22:07:06] 10Deployment-Systems: Make make-wmf-branch able to branch extensions with replaced substring of the version of mediawiki being branched - https://phabricator.wikimedia.org/T51392#1655043 (10greg) We'll need the changes to make-wmf-branch done pretty soon, we're only a week away from 1.27 branching.... [22:08:34] 10Deployment-Systems: Make make-wmf-branch able to branch extensions with replaced substring of the version of mediawiki being branched - https://phabricator.wikimedia.org/T51392#1655049 (10greg) Question: will the wiki page names change as well, I presume? eg: https://www.mediawiki.org/wiki/MediaWiki_1.27/wmf1... [22:09:14] !log deployment-eventlogging02:/var full due to massive eventlogging_processor-server-side-0.log.1 ("(MainThread) Unable to process" errors) [22:10:28] oh logmsgbot? [22:10:40] good thing bd's tool works: https://tools.wmflabs.org/sal/log/AU_igRQw1oXzWjit5mYu [22:11:02] can i delete that log file? [22:11:27] 10Beta-Cluster, 10Analytics-EventLogging, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Tom Waits: Beta Cluster EventLogging data is disappearing? - https://phabricator.wikimedia.org/T112926#1655055 (10greg) >>! In T112926#1650405, @Ottomata wrote: > Please use that ins... [22:11:44] marxarelli: dunno, I just pinged ottomata, but, he's probably not online [22:11:54] marxarelli: ping people in -analytics, I guess [22:12:35] k [22:14:06] * greg-g just pinged milimetric [22:14:08] in there [22:18:32] !log stopping eventlogging services on deployment-eventlogging02 and truncating logs [22:31:53] qa-morebots, how’s it going? [22:31:53] I am a logbot running on tools-exec-1203. [22:31:53] Messages are logged to https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL. [22:31:53] To log a message, type !log . [22:32:25] greg-g: Is ^ the bot you meant? [22:32:33] andrewbogott: yep! [22:32:46] ok — I restarted it. Probably lost to a netsplit or something [22:33:07] !log test test [22:33:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:34:22] !log we missed log entries on the wikitech log from the 15th, see https://tools.wmflabs.org/sal/releng?p=0&q=&d=2015-09-18 [22:34:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:37:51] RECOVERY - Free space - all mounts on deployment-eventlogging02 is OK: OK: All targets OK [22:40:09] ebernhardson: these jobrunner errors look cirrus related and new: https://logstash-beta.wmflabs.org/#dashboard/temp/AU_inR5Aa1EjumVd_Tx7 [22:41:12] greg-g: hmm, ok looks like flow might have changed something. i'll poke [22:44:23] for those playing along at home: https://phabricator.wikimedia.org/T113106 [22:46:44] * greg-g contemplates announcing the #wikimedia-log-errors tasks in... somewhere [22:46:52] having wikibugs* [22:48:58] 3Scap3: Support smooth transitions from Trebuchet managed deploys - https://phabricator.wikimedia.org/T113107#1655136 (10dduvall) 3NEW [22:49:56] (03CR) 10Dduvall: "I created a new task for it." [tools/scap] - 10https://gerrit.wikimedia.org/r/238839 (https://phabricator.wikimedia.org/T109514) (owner: 10Dduvall) [22:50:34] 10Deployment-Systems, 3Scap3, 5Patch-For-Review: Scap3 should support rollback - https://phabricator.wikimedia.org/T109514#1655146 (10dduvall) [22:50:36] 3Scap3: Support smooth transitions from Trebuchet managed deploys - https://phabricator.wikimedia.org/T113107#1655145 (10dduvall) [22:52:11] I like your thinking marxarelli "Bikeshedding: What about calling it legacy_dir? That way we get annoyed every time we see the word "legacy" and feel motivated to migrate the remaining Trebuchet repos to use our new tool. :)" [22:52:37] 10Deployment-Systems, 3Scap3: Support smooth transitions from Trebuchet managed deploys - https://phabricator.wikimedia.org/T113107#1655150 (10greg) [22:57:00] greg-g: how to responsibly employ microaggressions [22:57:09] word [22:57:15] mind games man [22:57:26] I knew you were management material [22:57:33] nooooooo! [22:57:39] I take it back! [23:47:21] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<22.22%)