[00:02:22] 10MediaWiki-Codesniffer: Add sniff that preg_quote should not be used with 1 argument - https://phabricator.wikimedia.org/T203312 (10Krinkle) >>! In T203312#4551429, @thiemowmde wrote: > Oh, wait. Using `preg_quote` with no argument is entirely valid: > ```lang=php > $regex = preg_quote( $searchString ); > preg_... [00:02:29] 10MediaWiki-Codesniffer: Add sniff that preg_quote should not be used with 1 argument - https://phabricator.wikimedia.org/T203312 (10Krinkle) p:05Triage>03Normal [00:11:47] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:34:33] (03CR) 10KartikMistry: [C: 031] "@hashar, can you look at this?" [integration/config] - 10https://gerrit.wikimedia.org/r/450508 (https://phabricator.wikimedia.org/T86930) (owner: 10Santhosh) [05:48:57] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10elukey) @Dzahn What are the steps missing to failover to phab1002? This task seems close to be able to finally do it :) [08:10:23] (03PS2) 10Hashar: Quibble jobs wipe src on completion [integration/config] - 10https://gerrit.wikimedia.org/r/457918 [08:32:57] zeljkof: where can I find the config for jobs like https://integration.wikimedia.org/ci/job/selenium-daily-beta-WikibaseLexeme/14/console in integration/config ? [08:33:47] addshore: ah, it's still WIP https://gerrit.wikimedia.org/r/c/integration/config/+/457882 [08:33:55] gotcha [08:36:00] * addshore notices SERVER_NAME is https://en.wikipedia.beta.wmflabs.org [08:36:09] * addshore also thinks wikibase is not loaded during the tests [08:36:11] * addshore stops looking :D [08:43:11] (03PS3) 10Hashar: Wipe src upon Quibble jobs completion [integration/config] - 10https://gerrit.wikimedia.org/r/457918 (https://phabricator.wikimedia.org/T202457) [09:10:30] (03PS4) 10Hashar: Wipe src upon Quibble jobs completion [integration/config] - 10https://gerrit.wikimedia.org/r/457918 (https://phabricator.wikimedia.org/T202457) [09:12:00] (03CR) 10Hashar: [C: 032] Wipe src upon Quibble jobs completion [integration/config] - 10https://gerrit.wikimedia.org/r/457918 (https://phabricator.wikimedia.org/T202457) (owner: 10Hashar) [09:12:56] (03CR) 10Hashar: [C: 032] Add abian to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/459572 (owner: 10Lucas Werkmeister (WMDE)) [09:14:13] (03Merged) 10jenkins-bot: Wipe src upon Quibble jobs completion [integration/config] - 10https://gerrit.wikimedia.org/r/457918 (https://phabricator.wikimedia.org/T202457) (owner: 10Hashar) [09:14:25] (03Merged) 10jenkins-bot: Add abian to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/459572 (owner: 10Lucas Werkmeister (WMDE)) [09:24:25] (03PS43) 10Hashar: Selenium daily tests for beta using Docker/wdio [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) [09:26:20] (03CR) 10Hashar: [C: 032] "The new jobs were no more generated since the job template was not applied to the jjb project." [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) (owner: 10Hashar) [09:28:20] (03Merged) 10jenkins-bot: Selenium daily tests for beta using Docker/wdio [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) (owner: 10Hashar) [09:34:54] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10hashar) Wi... [09:35:49] (03PS2) 10Hashar: Create selenium-daily-beta Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/457882 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:37:02] (03PS3) 10Hashar: Create selenium-daily-beta Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/457882 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:38:18] !log sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/selenium-daily-beta-Wikibase*' # T188742 [09:38:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:38:22] T188742: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 [09:51:16] (03CR) 10Hashar: "For Wikibase https://integration.wikimedia.org/ci/job/selenium-daily-beta-Wikibase/16/console" [integration/config] - 10https://gerrit.wikimedia.org/r/457882 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:51:27] (03CR) 10Hashar: [C: 04-1] Create selenium-daily-beta Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/457882 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:52:23] 10MediaWiki-Codesniffer: Add sniff that preg_quote should not be used with 1 argument - https://phabricator.wikimedia.org/T203312 (10thiemowmde) `/…/` is the standard, and used in the majority of cases – //unless// there is a good reason to free the slash from being a special character. A typical example is some... [09:58:37] (03PS2) 10Hashar: Create selenium-daily-beta-MediaWiki Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/457881 (https://phabricator.wikimedia.org/T185011) (owner: 10Zfilipin) [09:59:55] (03CR) 10Hashar: "Rebased and deployed, it passes just fine." [integration/config] - 10https://gerrit.wikimedia.org/r/457881 (https://phabricator.wikimedia.org/T185011) (owner: 10Zfilipin) [10:00:08] zeljkof: are you around? :) [10:00:16] zeljkof: selenium-daily-beta-MediaWiki is green I am willing to +2 the patch [10:00:28] hashar: I'm around! [10:00:32] the last thing is that it spam us directly, I thought about using qa-alerts@lists.wikimedia.org betacluster-alerts@list.wikimedia.org instead [10:00:33] +2 it* [10:00:48] oh, I forgot about that [10:00:57] I have got the patch ready :] [10:01:03] I'm not subscribed to any of those lists, so feel free to to leave me in [10:01:05] (03PS3) 10Hashar: Create selenium-daily-beta-MediaWiki Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/457881 (https://phabricator.wikimedia.org/T185011) (owner: 10Zfilipin) [10:01:18] zeljkof: https://gerrit.wikimedia.org/r/#/c/integration/config/+/457881/3/jjb/mediawiki.yaml [10:01:43] hashar: you should put me back in [10:01:58] I'm not subscribed to any -alerts lists [10:02:02] (03PS4) 10Hashar: Create selenium-daily-beta-MediaWiki Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/457881 (https://phabricator.wikimedia.org/T185011) (owner: 10Zfilipin) [10:02:03] adding you [10:02:08] thanks! [10:02:21] deploying and merging [10:02:34] (03CR) 10Zfilipin: [C: 031] Create selenium-daily-beta-MediaWiki Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/457881 (https://phabricator.wikimedia.org/T185011) (owner: 10Zfilipin) [10:02:39] looks good! [10:02:42] thanks! [10:02:50] (03CR) 10Hashar: [C: 032] "Updated to email Zeljko, qa-alerts and betacluster-alerts lists." [integration/config] - 10https://gerrit.wikimedia.org/r/457881 (https://phabricator.wikimedia.org/T185011) (owner: 10Zfilipin) [10:04:07] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [10:04:29] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10hashar) [10:04:31] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [10:06:55] (03PS1) 10Hashar: jjb: npm-install-selenium macro is no more used [integration/config] - 10https://gerrit.wikimedia.org/r/459756 (https://phabricator.wikimedia.org/T188742) [10:07:32] (03Merged) 10jenkins-bot: Create selenium-daily-beta-MediaWiki Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/457881 (https://phabricator.wikimedia.org/T185011) (owner: 10Zfilipin) [10:08:11] (03PS1) 10Hashar: jjb: inject-experimental-extension macro is no more used [integration/config] - 10https://gerrit.wikimedia.org/r/459757 [10:09:13] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [10:09:28] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811 (10hashar) [10:09:33] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [10:11:30] (03CR) 10Hashar: [C: 032] "Noop in JJB." [integration/config] - 10https://gerrit.wikimedia.org/r/459756 (https://phabricator.wikimedia.org/T188742) (owner: 10Hashar) [10:11:58] (03CR) 10Hashar: [C: 032] "Noop in JJB." [integration/config] - 10https://gerrit.wikimedia.org/r/459757 (owner: 10Hashar) [10:13:29] (03Merged) 10jenkins-bot: jjb: npm-install-selenium macro is no more used [integration/config] - 10https://gerrit.wikimedia.org/r/459756 (https://phabricator.wikimedia.org/T188742) (owner: 10Hashar) [10:14:09] (03Merged) 10jenkins-bot: jjb: inject-experimental-extension macro is no more used [integration/config] - 10https://gerrit.wikimedia.org/r/459757 (owner: 10Hashar) [10:27:24] 10MediaWiki-Codesniffer: Disallow use of @private and @protected using MediaWiki.Commenting.FunctionAnnotations.UnrecognizedAnnotation - https://phabricator.wikimedia.org/T203980 (10thiemowmde) Personally, I believe a `@private` annotation is a hint at a suboptimal architecture. However, having a `@private` anno... [10:33:26] o/ zeljkof [10:33:30] when did https://integration.wikimedia.org/ci/job/mwselenium-quibble-docker/422/console jobs start running? [10:33:46] just curious as they seem to be failing for master of WikibaseLexeme :9 [10:34:12] addshore for anything with quibble in the name, you should ask hashar :D [10:45:51] ack :D [10:59:20] addshore: a while ago ? :) [10:59:46] I think we found the thing that broke it! [10:59:57] addshore: is that for Wikibase? [11:00:27] for Wikibaase i did the switch 8 days ago https://gerrit.wikimedia.org/r/#/c/integration/config/+/457575/ [11:02:02] addshore: and WikibaseLexeme via https://gerrit.wikimedia.org/r/#/c/integration/config/+/457545/ 8 days ago [11:02:17] gotcha! [11:02:32] addshore: and I am 101% sure the job passed when I did the migration [11:02:50] meanwhile https://integration.wikimedia.org/ci/job/mwext-donationinterfacecore-REL1_27-zend56-jessie/1221/console [11:02:52] SUCCESS [11:02:54] I am such a hacker [11:03:23] ah no [11:05:08] :D [11:06:04] (03PS1) 10Hashar: Add experimental DonationInterface job mediawiki [integration/config] - 10https://gerrit.wikimedia.org/r/459767 (https://phabricator.wikimedia.org/T203084) [11:06:19] (03CR) 10Hashar: [C: 032] Add experimental DonationInterface job mediawiki [integration/config] - 10https://gerrit.wikimedia.org/r/459767 (https://phabricator.wikimedia.org/T203084) (owner: 10Hashar) [11:07:50] (03Merged) 10jenkins-bot: Add experimental DonationInterface job mediawiki [integration/config] - 10https://gerrit.wikimedia.org/r/459767 (https://phabricator.wikimedia.org/T203084) (owner: 10Hashar) [12:20:07] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-MModell, 10User-Ryasmeen: 404 on workboard for an existing project (due to custom filter applied which did not exist in database) - https://phabricator.wikimedia.org/T199207 (10Deskana) This happened again on the #VisualEditor workboard last Wednes... [13:28:25] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-DonationInterface, 10Patch-For-Review: Migrate DonationInterface CI job to Docker - https://phabricator.wikimedia.org/T203084 (10hashar) So that will be solved by bumping wikimedia/composer-merge... [13:44:11] 10MediaWiki-Codesniffer: Add sniff that preg_quote should not be used with 1 argument - https://phabricator.wikimedia.org/T203312 (10Anomie) >>! In T203312#4573501, @thiemowmde wrote: > `/…/` is the standard, and used in the majority of cases – //unless// there is a good reason to free the slash from being a spe... [13:49:10] PROBLEM - SSH on integration-slave-docker-1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:23] Hey, maybe this has been asked by others somewhere and I haven't seen it, but just asking: https://fixcopyright.wikimedia.org/ has bad or lacking translations in some cases, where new translations have been made but not yet made it to the page, and the vote in the parliament is tomorrow. This is part of l10nupdate and the realistic reply is "nope, sorry, you won't be able to use the campaign", or is there some way around it? [13:59:57] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-DonationInterface, 10Patch-For-Review: Migrate DonationInterface CI job to Docker - https://phabricator.wikimedia.org/T203084 (10hashar) And to confirm, I gave it a try with: | mediawiki/core |... [14:05:15] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10hashar) [14:05:17] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, and 2 others: Create selenium-daily-beta-MediaWiki daily Jenkins job - https://phabricator.wikimedia.org/T185011 (10hashar) 05Open>03Resolved Pass https://in... [14:07:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): mediawiki-quibble docker jobs fails due to disk full - https://phabricator.wikimedia.org/T202457 (10hashar) The Quibble jobs now delete the `src` directory... [14:17:35] Yippee, build fixed! [14:17:36] Project mediawiki-core-doxygen-docker build #1477: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/1477/ [14:20:46] PROBLEM - Puppet errors on deployment-certcentral03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:24:18] (03PS1) 10Hashar: Dont wipe src for mediawiki-core-doxygen-docker [integration/config] - 10https://gerrit.wikimedia.org/r/459797 [14:24:32] (03CR) 10Hashar: [C: 032] Dont wipe src for mediawiki-core-doxygen-docker [integration/config] - 10https://gerrit.wikimedia.org/r/459797 (owner: 10Hashar) [14:26:29] (03Merged) 10jenkins-bot: Dont wipe src for mediawiki-core-doxygen-docker [integration/config] - 10https://gerrit.wikimedia.org/r/459797 (owner: 10Hashar) [14:35:46] RECOVERY - Puppet errors on deployment-certcentral03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:54:48] hashar: how does one get secrets into WMF ci? [14:58:58] addshore: fill a task and we sort it out for you :] [14:59:15] addshore: more seriously, Jenkins has a system to hold secret: the credential store [14:59:24] so we would generate some secret [14:59:27] it relates to https://phabricator.wikimedia.org/T192006 and wanting to basically hit a travis api endpoint after a merge in a repo [14:59:29] store them in jenkins credential store [14:59:47] and optionally store the password in the releng shared store (gpg / git based) [15:00:07] ah [15:00:24] well the repository is replicated to github isn't it ? Cant github trigger the travis thing from there? [15:03:35] also we have a new facny system to define docker images, build them and run tests against them [15:03:38] Project mediawiki-core-code-coverage-docker build #3755: 04FAILURE in 3 min 37 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/3755/ [15:03:41] with publishing to the wikimedia docker registry [15:04:02] hashar: yes, but not for a seperate repo [15:04:32] hashar: also yes, moving the docker images to gerrit could also be a cool plan [15:04:46] sounds like the sort of things we need some docs for :D [15:05:33] ejegg: hello! I found out the issue I had with DonationInterface and amzn/login-and-pay-with-amazon-sdk-php . The short story is we have to upgrade wikimedia/composer-merge-plugin from 1.3.1 to 1.4.1 :) I have no clue how to do that for mediawiki/vendor though (ref https://phabricator.wikimedia.org/T203084 ) [15:06:10] thanks hashar! [15:06:19] I'll try to figure it out from here [15:06:39] ejegg: I am hitting a wall at updating the package in mediawiki/vendor fundraising/REL1_27 :-\ composer keeps bringing unrelated stuff [15:07:07] addshore: what is the repo ? Surely it could be made to have the fancy stuff we are working on (blubber + pipeline) [15:07:40] ejegg: the bad patch I had is https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/459792/ ;D [15:07:47] the repo we need to trigger the build on? or the repo triggering the build? [15:08:06] it is the wdqs frontend, I actually mentioned it might be another good candidate to deploy with the new system to thcipriani [15:08:08] addshore: hmm. Both? :] [15:08:29] so yeah that is my idea, lets implement it with the new system [15:08:29] the other one it https://github.com/wmde/wikibase-docker which currently holds the wikibase docker images [15:08:42] hashar: I like the sound of that.... [15:09:00] it is working for mathoid at least [15:09:18] we want to get some new experience with the new system too as we want to build and deploy a vue JS SSR renderer for wikidata.org in the next 12 months [15:09:41] assuming wikidata-query-gui is the frontend and its is a javascript app of some sort, it is probably easy to convert it [15:09:51] yup, it is all js :) [15:09:57] 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline: Update Debian package of Blubber (0.5.0-1) - https://phabricator.wikimedia.org/T203121 (10thcipriani) Poking this for ETA. This one should unblock us on graphoid as well as a add a builder to help support generic jobs in CI... [15:10:07] are there docs? :P [15:10:11] speaking of the devil [15:10:51] hehe [15:10:54] addshore: I don't think we generate / publish doc [15:11:00] there might be some in the blubber.git repo thugh [15:11:32] then there is a jenkins pipeline job to clone the repo, run blubber to generate the Dockerfile and then create the image and publish it [15:12:23] thcipriani / marxarelli would be better entry points :] [15:13:04] there are the beginning of docs [15:13:09] https://wikitech.wikimedia.org/wiki/Blubber [15:13:13] covers blubber usage [15:13:17] wheeee, i wana give it a go :D [15:13:25] would give me an excuse to write another blog post too [15:13:34] \o/ [15:13:50] :D [15:13:52] I am late in writting the Quibble in summer blog post bah [15:14:37] blog posts are hard [15:14:44] there is never enough time for them [15:14:44] if you can get the: blubber .pipeline/blubber.yaml test | docker build -t test-thing && docker run test-thing to work, there is an experimental blubber generic test runner in CI [15:15:49] service-pipeline-test-only IIRC [15:16:52] ah, no, blubber-test! https://integration.wikimedia.org/ci/job/blubber-test/ [15:16:58] ooooh [15:17:02] *reads the wiki page* [15:18:10] mathoid is the only example in the wild currently: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/mathoid/+/master/.pipeline/blubber.yaml [15:19:03] RECOVERY - SSH on integration-slave-docker-1004 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u6 (protocol 2.0) [15:22:53] SMalyshev: how is wikidata/query/gui currently deployed? O_o, I don't see it in puppet anywhere at all [15:25:10] PROBLEM - SSH on integration-slave-docker-1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:27:48] haha, thcipriani, there are windows binaries for blubber? :P [15:28:36] untested, but cross-compiling docs for go seemed straight-forward, so yes :) [15:28:38] I mean, I totally would have taken a docker image for it, but woo for convenience (yes im evil and run windows) [15:29:22] there's also "blubberoid" currently running in toolforge [15:29:30] that's a bit newer than what's deployed via apt though [15:30:00] so you can dump dockerfiles via blubber through curl [15:30:30] v3 is unsupported apparently =o [15:30:38] I'll switch to v2 as is in the current mathoid file [15:31:31] yeah, v3 is what blubberoid is using, v2 is what's in integration/apt currently IIRC [15:32:02] Soon™ https://phabricator.wikimedia.org/T203121 [15:36:25] what an exciting afternoon [15:36:39] my sister is going into labour and I'm blubberizing things [15:37:02] 10Continuous-Integration-Config, 10Wikidata: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Lucas_Werkmeister_WMDE) [15:38:46] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Addshore) [15:38:49] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Lucas_Werkmeister_WMDE) [15:41:14] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Addshore) Looking at core merges recently it must be one of these 2 patches: Introduce RevisionRecord::isReadForInsertion https://ger... [15:41:25] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Addshore) p:05Triage>03High [15:41:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, and 2 others: Popups daily jobs currently unusable - https://phabricator.wikimedia.org/T203591 (10zeljkofilipin) I'm not sure what the problem i... [15:46:51] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Addshore) p:05High>03Unbreak! [15:50:52] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Lucas_Werkmeister_WMDE) The `NameTableAccessException` feels like it could be happening because the `content_models` table is purged b... [15:51:28] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Addshore) >>! In T204065#4574491, @Lucas_Werkmeister_WMDE wrote: > The `NameTableAccessException` feels like it could be happening bec... [15:53:03] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, and 2 others: Create selenium-daily-beta-MediaWiki daily Jenkins job - https://phabricator.wikimedia.org/T185011 (10zeljkofilipin) [16:00:21] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Pablo-WMDE) So @Addshore seems to be on it. [16:02:26] >> /srv/service/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs.exe: 1: /srv/service/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs.exe: MZ�♥♦���@@☺♫▼�♫�: not found 0 [ '/srv/service/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs.exe: 1: /srv/service/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs.exe: MZ�\u0003\u0004���@@\u0001\u000e\u001f�\u000e�: not found', [16:02:28] thcipriani: :P [16:03:13] I guess that shouldnt be in the tests for the wdqs-gui while running in a linux docker container xD [16:03:57] tbh, i probably shouldnt be testing this blubber thing on a windows how anyway as everything will have the wrong filer permissions [16:04:00] guessi should do it on labs... [16:04:36] file permissions shouldn't matter too much, everything happens in the container, there's nothing really left over [16:04:58] I wonder why it is looking for a pahnton js exe though [16:05:30] maybe I should delete my local node_modules file before running... docker build -t wdqs-blubber-test . && docker run wdqs-blubber-test [16:05:36] right :) [16:05:39] I was just about to say [16:06:09] is removing the node_modules dir a requirement? :( [16:06:09] there's a step where docker copies in the local directory, it's expecting it to be clean [16:06:20] gotcha [16:07:22] we should do something smarter there. We've mostly been testing in CI, and we don't do a lot of local dev on the repos we're testing with so it hasn't come up. [16:07:30] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Addshore) a:03daniel [16:08:28] although that's strange. I thought using NODE_PATH might be enough. [16:08:36] I guess NODE_PATH is just a fallback? [16:09:56] it might be cool for it to delete everything in the .gitignore file actually [16:10:18] for example in the wdqs-gui repo I also have a build directory etc, that I guess I'll have to remove in order to test the blubber build [16:10:47] urgfff, and things like the .idea directory for my IDE :( [16:17:48] Yippee, build fixed! [16:17:48] Project mediawiki-core-doxygen-docker build #1478: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/1478/ [16:20:02] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [16:24:28] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Easy, 10User-zeljkofilipin: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [16:28:11] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:30:31] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:37:32] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:38:20] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:43:42] addshore: i would just add node_modules to .dockerignore [16:43:56] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilip... [16:44:34] that will more or less be a requirement for building images for node projects with blubber, since you don't want anything that was natively compiled to end up in the container image [16:44:43] (in case the architectures are different) [16:47:12] also, blubber currently installs the node_modules to a location outside the project root (for strange reasons that may no longer apply to the primary use case of blubber :) ) [16:47:42] thcipriani and i have talked about refactoring that [16:48:54] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10Patch-For-Review, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Lucas_Werkmeister_WMDE) Investigation is still going on, but it looks like reverting the MCR migration stage wou... [16:49:36] marxarelli: thats a great idea.. :P [16:50:58] *tries again* [16:55:17] (03CR) 10Zfilipin: "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/457882 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [17:02:27] (03CR) 10Umherirrender: [C: 031] "Ic3485c0d09b7e3a66f8dda6f1184cfed599f84c1 is now merged" [integration/config] - 10https://gerrit.wikimedia.org/r/457109 (https://phabricator.wikimedia.org/T203364) (owner: 10MGChecker) [17:26:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review, and 2 others: Popups daily jobs currently unusable - https://phabricator.wikimedia.org/T203591 (10Jdlrobson) >>! In T203591#4574473, @zeljkofili... [17:28:11] PROBLEM - SSH on integration-slave-docker-1015 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:35] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: Scap should use Eval.Jit=1 when calling rebuildLocalisationCache.php via HHVM - https://phabricator.wikimedia.org/T203680 (10thcipriani) [17:34:11] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes on HHVM (rather than ~5 on PHP 5) - https://phabricator.wikimedia.org/T191921 (10thcipriani) [17:36:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins build log and results to ElasticSearch - https://phabricator.wikimedia.org/T78705 (10dduvall) @EBernhardson, I'm testing the logstash publisher to see if it'll at... [17:39:57] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: Scap should use Eval.Jit=1 when calling rebuildLocalisationCache.php via HHVM - https://phabricator.wikimedia.org/T203680 (10thcipriani) I've merged a patch to scap to allow setting `php_version` in `scap.cfg` that will be used... [17:43:24] 10Release-Engineering-Team, 10Patch-For-Review: tools.wmflabs.org/versions caching - https://phabricator.wikimedia.org/T202734 (10bd808) 05Open>03Resolved a:03bd808 I reverted the cache busting code. Please do reopen if the problem is seen again. [17:51:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins build log and results to ElasticSearch - https://phabricator.wikimedia.org/T78705 (10EBernhardson) That is still the address for our testing servers, yes. The serv... [17:53:03] RECOVERY - SSH on integration-slave-docker-1015 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u6 (protocol 2.0) [17:59:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): maintenance-disconnect-full-disks gets stuck - https://phabricator.wikimedia.org/T204077 (10thcipriani) [17:59:12] PROBLEM - SSH on integration-slave-docker-1015 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): maintenance-disconnect-full-disks gets stuck - https://phabricator.wikimedia.org/T204077 (10thcipriani) p:05Triage>03Normal [18:18:24] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T191066 (10Jdforrester-WMF) Can we declare this Resolved? The wmf.20 is everywhere… [18:20:07] 10MediaWiki-Codesniffer: Add sniff that preg_quote should not be used with 1 argument - https://phabricator.wikimedia.org/T203312 (10Krinkle) >>! In T203312#4573501, @thiemowmde wrote: > Sure, we could still enforce the second parameter in `preg_quote` to be `'/'`, even if brackets are used as delimiters. [..]... [18:20:39] 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline: Update Debian package of Blubber (0.5.0-1) - https://phabricator.wikimedia.org/T203121 (10akosiaris) >>! In T203121#4574382, @thcipriani wrote: > Poking this for ETA. > > This one should unblock us on graphoid as well as a a... [18:20:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins build log and results to ElasticSearch - https://phabricator.wikimedia.org/T78705 (10dduvall) >>! In T78705#4574910, @EBernhardson wrote: > That is still the addre... [18:20:55] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T191066 (10Krinkle) [18:20:58] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T191066 (10greg) 05Open>03Resolved Indeed. [18:21:22] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T191066 (10Krinkle) T203566 is still being worked on but, yeah, we have rolled out, and T203566 explicitly prevents us from rolling back as well (we tried..), so fine to... [18:25:54] 10MediaWiki-Codesniffer: Add sniff that preg_quote should not be used with 1 argument - https://phabricator.wikimedia.org/T203312 (10Anomie) Note that preg_quote quotes `!` by default, which might explain its use as a delimiter. Also, just because a regex happens to include preg_quote doesn't mean that leaning t... [18:32:08] 10Continuous-Integration-Infrastructure, 10MediaWiki-Cache, 10Patch-For-Review, 10Technical-Debt: BagOStuff should detect obsolete serialization or an unserialization resulting in a "wrong" object - https://phabricator.wikimedia.org/T156541 (10Krinkle) Not a sub task per T203566#4564404. T203566 isn't req... [18:32:17] 10Continuous-Integration-Infrastructure, 10MediaWiki-Cache, 10Patch-For-Review, 10Technical-Debt: BagOStuff should detect obsolete serialization or an unserialization resulting in a "wrong" object - https://phabricator.wikimedia.org/T156541 (10Krinkle) [18:32:26] 10Continuous-Integration-Infrastructure, 10MediaWiki-Cache, 10Technical-Debt: BagOStuff should detect obsolete serialization or an unserialization resulting in a "wrong" object - https://phabricator.wikimedia.org/T156541 (10Krinkle) [18:32:28] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10Patch-For-Review, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10greg) p:05Unbreak!>03High So after those reverts it's no longer broken, correct? Resetting priority based on... [18:38:14] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10Patch-For-Review, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Jdforrester-WMF) >>! In T204065#4575049, @greg wrote: > So after those reverts it's no longer broken, correct? R... [18:42:35] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10Patch-For-Review, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10greg) Sorry, I couldn't tell from this task what the issue is; is it CI? If so in what way? Is it the change/how... [18:46:03] 10Continuous-Integration-Config, 10Multi-Content-Revisions, 10Wikidata, 10Patch-For-Review, 10User-Addshore: Wikibase CI broken (database errors) - https://phabricator.wikimedia.org/T204065 (10Jdforrester-WMF) Switching MCR in master to `read-new` broke a bunch of the Wikibase-related extensions' unit te... [19:00:07] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T191066 (10hashar) I have left it open indicating that some issues where still being working on. But yeah the stack looks more or less fine now. [19:08:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins build log and results to ElasticSearch - https://phabricator.wikimedia.org/T78705 (10hashar) T140257 has the history. In short, `contint1001` is in the production... [19:15:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins build log and results to ElasticSearch - https://phabricator.wikimedia.org/T78705 (10EBernhardson) >>! In T78705#4575134, @hashar wrote: > If that is the case, an... [19:38:55] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes on HHVM (rather than ~5 on PHP 5) - https://phabricator.wikimedia.org/T191921 (10hashar) Seems scap learned to set the `PHP` environment variable for mwscript. I guess we can now try `hhvm -d hh... [19:39:22] thcipriani: you can probably try to have scap to use PHP=hhvm -d hhvm.stats.enable_hot_profiler=0 now :] [19:40:01] Timo says the hotprofiler is for Xenon profiling data and we do not profile CLI scripts https://phabricator.wikimedia.org/T191921#4558120 [19:42:42] marxarelli: the Quibble jobs are now cleaning out ./src on build completion. That should lower the disk pressure on the multiple executors slaves \o/ [19:48:20] great! [19:50:07] 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review: Rebuild quibble images for Chrome 69 and Firefox 60 - https://phabricator.wikimedia.org/T203902 (10hashar) I have split the updates in different changes. I will probably rebuild all images at once then update the jobs one per one and monit... [19:53:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): mediawiki-quibble docker jobs fails due to disk full - https://phabricator.wikimedia.org/T202457 (10hashar) p:05High>03Normal The 90 Quibble jobs left b... [19:55:34] (03CR) 10Hashar: "I think the jobs got all reverted when I have updated the Quibble jobs for some other tasks." [integration/config] - 10https://gerrit.wikimedia.org/r/457070 (https://phabricator.wikimedia.org/T203181) (owner: 10Legoktm) [19:56:39] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review, 10Regression: Quibble MariaDB should use a tmpfs as a datadir - https://phabricator.wikimedia.org/T203181 (10hashar) From https://gerrit.wikimedia.org/r/#/c/integration/config/+/457070/: Quibbl... [19:56:46] 10Continuous-Integration-Infrastructure (shipyard), 10Patch-For-Review: Rebuild quibble images for Chrome 69 and Firefox 60 - https://phabricator.wikimedia.org/T203902 (10hashar) [19:56:48] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review, 10Regression: Quibble MariaDB should use a tmpfs as a datadir - https://phabricator.wikimedia.org/T203181 (10hashar) [19:57:38] 10MediaWiki-Codesniffer: Add sniff that preg_quote should not be used with 1 argument - https://phabricator.wikimedia.org/T203312 (10Umherirrender) The special regular expression characters are: . \ + * ? [ ^ ] $ ( ) { } = ! < > | : - From http://www.php.net/manual/en/function.preg-quote.php Using the taint-ch... [19:58:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-Addshore: Add some more m4executor docker slaves for Jenkins - https://phabricator.wikimedia.org/T201972 (10hashar) {T202457} should be less of an issue now, the Quibble based jobs clear out `src` which... [21:00:37] PROBLEM - SSH on integration-slave-docker-1025 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Free up LVM extents for Docker devicemapper on new Jenkins Agents - https://phabricator.wikimedia.org/T203842 (10dduvall) a:03dduvall [21:38:08] 10Scap (Scap3-MediaWiki-MVP), 10MediaWiki-Configuration: SCAP should not rely on extension-list, instead pass --extension-dir to mergeMessageFileList.php - https://phabricator.wikimedia.org/T125678 (10Krinkle) [21:38:38] 10Scap (Scap3-MediaWiki-MVP), 10MediaWiki-Configuration: SCAP should not rely on extension-list, instead pass --extension-dir to mergeMessageFileList.php - https://phabricator.wikimedia.org/T125678 (10Krinkle) [21:39:08] !log cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459850 on integration-puppetmaster01 [21:39:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:49:20] !log removing unresponsive jenkins node integration-slave-docker-1025 [21:49:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:52:28] PROBLEM - Host integration-slave-docker-1025 is DOWN: CRITICAL - Host Unreachable (10.68.22.96) [22:05:53] !log launching replacement instance integration-slave-docker-1028 [22:05:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:28:26] PROBLEM - Puppet errors on integration-slave-docker-1028 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:40:33] !log deleting integration-slave-docker-1028 in favor of trying a stretch instance [22:40:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:42:34] PROBLEM - Host integration-slave-docker-1028 is DOWN: CRITICAL - Host Unreachable (10.68.19.55) [22:43:16] !log launching m1.xlarge integration-slave-docker-1029 using stretch image [22:43:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:54:58] PROBLEM - Puppet errors on integration-slave-docker-1029 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [22:59:19] thcipriani: ^ negative. no zuul or docker-ce package from the looks of it [22:59:35] huh, what do we need zuul for? [22:59:42] zuul-cloner? [22:59:45] going back to a jessie instance. i guess we'll have to figure out a different way to configure docker [23:00:01] i think? i'm just looking at the puppet output atm [23:01:05] for docker-ce: https://phabricator.wikimedia.org/T190585 [23:04:55] I do wonder if we need zuul-cloner, even, anymore. There is definitely a zuul-clone docker image. [23:13:20] thcipriani: re: docker-ce, looks like it might have been a missing dependency between the thirdparty/ci apt config and the package resource. it installs now but fails to start [23:13:33] zuul is still missing though [23:14:30] but yeah, i agree that ditching zuul-cloner might be possible/preferable. one less dependency to worry about on ci nodes [23:26:15] PROBLEM - Puppet errors on deployment-certcentral-testclient03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:27:47] hrm so should moving integration agents to stretch block changing over the docker storage driver? or should we stick with overlayfs and move where images are stored? [23:28:55] thcipriani: it appears to block it, yeah [23:29:24] and i'm running into other issues trying to configure docker even on stretch [23:29:40] sounds like we need to reopen https://phabricator.wikimedia.org/T190585 [23:29:41] e.g. it wants its own volume group [23:30:40] thcipriani: actually, looking again i believe it's just the additional /etc/docker/daemon.json config that's causing it to fail [23:30:45] i think the package is ok [23:30:51] ah cool [23:31:28] but using the dm storage driver is going to be tricky on labs instances i think [23:31:53] we may be better off just creating a second logical volume and configuring docker to put its overlayfs there [23:32:22] unless we're willing to futz with re-partitioning /dev/vda :/ [23:33:25] we haven't had any problems with overlayfs aside from sharing space with the OS partition [23:33:41] or leaving /srv on the / partition, also an option perhaps [23:33:49] now that src directories are cleaned up [23:34:14] if we did that, we could give docker the entire vd volume group [23:34:26] although it would be interesting to see if we do get performance gains from dm [23:34:46] it would [23:35:11] i'd rather solve for stability first, however [23:35:23] I don't know if 20GB is enough for workspaces and system files [23:35:42] seems like workspaces by themselves were hovering around 12GB [23:35:44] last I checked [23:35:58] that would be tight then [23:36:14] we could try it on an instance, keep an eye on it and see what happens [23:36:46] since /src is now cleaned, I would imagine that is now reduced quite a bit [23:37:28] CI is constantly moving [23:37:41] it sure is! [23:39:01] might try an m1.medium with 1 executor and not assigning the /srv partition and configuring device mapper to use the volume group [23:39:14] with 1 executor workspace space is bound to be small [23:39:26] now that workspaces are cleaned post-build [23:41:11] i'm leaning towards just keeping overlayfs and using an additional lv for it [23:41:24] i think that'll be a simpler setup for now [23:41:45] might even be able to mount it at /var/lib/docker :) [23:45:43] neat :)