[00:01:22] my main concern with jjb is I think we've sort of forked a point-in-time from upstream and they've since moved on [00:01:40] (ditched jenkins entirely IIRC) [00:01:53] they're still using jjb without jenkins? [00:02:13] how do you use *jenkins* job builder without jenkins... [00:02:19] eh...they're using something akin to jjb that builds ansible playbooks, IIRC [00:02:27] (I have not looked too deeply here) [00:03:08] my very rough understanding of their setup is basically: zuul (version 3), nodepool, and ansible [00:03:16] we've also effectively forked zuul by sticking with v2 right? [00:03:26] for the time being, yeah [00:03:46] we have some time dedicated at our upcoming offsite (in december) to talk about what we're going to do [00:03:56] we should probably re-evaluate the entire setup we have and what upstream Gerrit is using for CI instead of continuing to blindly follow openstack because I think following them down nodepool was a giant mistake [00:03:58] great [00:04:12] we haven't made a conscious decision to fork zuul, that's just what we're currently doing. [00:04:17] * legoktm nods [00:04:52] yeah, nodepool only makes sense if you've got a ton of openstack instances all donating space for your [00:04:54] *you [00:04:55] legoktm upstream use jenkins and gerritforge runs there ci [00:04:59] which uses scripts [00:05:02] which is here: [00:05:22] legoktm https://github.com/GerritCodeReview/gerrit-ci-scripts/tree/master/jenkins [00:05:28] not one openstack that you hammer into the ground [00:05:40] main scripts your wanting to look at is https://github.com/GerritCodeReview/gerrit-ci-scripts/blob/master/jenkins/gerrit-verifier-change.groovy [00:05:46] https://github.com/GerritCodeReview/gerrit-ci-scripts/blob/master/jenkins/gerrit-verifier-flow.groovy [00:05:52] https://github.com/GerritCodeReview/gerrit-ci-scripts/blob/master/jenkins/gerrit-verifier-postbuild.groovy [00:06:05] https://github.com/GerritCodeReview/gerrit-ci-scripts/blob/master/jenkins/gerrit-verifier.yaml [00:06:58] lolol [00:07:41] paladox: interesting, I'll poke around in my copious free time [00:07:48] :) [00:07:53] legoktm it dosen't use ssh [00:08:00] since google disabled ssh for gerrit-review [00:08:05] so it runs on a cron [00:08:13] and you can *decide* how many jobs run [00:08:36] so at the hackathon next month they will up the amout of jobs that can run at once (i think) [00:28:39] 10Release-Engineering-Team (Kanban), 10Wikimedia-Technical-Conference-2018, 10User-greg: Wikimedia Technical Conference 2018 Session - How do we work together? - https://phabricator.wikimedia.org/T206064 (10debt) {F26778133} {F26778132} {F26778131} {F26778130} {F26778129} {F26778128} {F26778127} {F267... [03:59:31] 10Continuous-Integration-Config, 10Quibble, 10Regression: MediaWiki PHPUnit tests no longer have "Test report" in jenkins with quibble - https://phabricator.wikimedia.org/T206227 (10Legoktm) [03:59:36] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Quibble should instruct PHPUnit to generate Junit files - https://phabricator.wikimedia.org/T207841 (10Legoktm) [04:00:47] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10Legoktm) [04:01:02] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10Legoktm) I +2'd https://gerrit.wikimedia.org/r/469544 into master, someone will need to backport it before the train should advance. [05:22:56] 10MediaWiki-Releasing, 10Growth-Team, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650 (10Legoktm) >>! In T196650#4689982, @Jdforrester-WMF wrote: >>>! In T196650#4689897, @MGChecker wrote: >> Wh... [07:08:12] 10Release-Engineering-Team (Kanban), 10Analytics-Tech-community-metrics, 10Code-Health: Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891 (10Quiddity) [07:42:13] (03CR) 10Hashar: [C: 032] "I have deployed the jobs!" [integration/config] - 10https://gerrit.wikimedia.org/r/469512 (owner: 10Hashar) [07:44:48] (03Merged) 10jenkins-bot: jjb: bump Quibble jobs to 0.0.28 [integration/config] - 10https://gerrit.wikimedia.org/r/469512 (owner: 10Hashar) [07:46:41] (03CR) 10Hashar: [C: 032] "170 jobs updated" [integration/config] - 10https://gerrit.wikimedia.org/r/469512 (owner: 10Hashar) [07:50:49] !log enabling puppet again on deployment-deploy01 . Was disabled by _joe_ for apache-fast-test hacking [07:50:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:05:02] (03PS1) 10Hashar: Record Junit files from Quibble PHPUnit run [integration/config] - 10https://gerrit.wikimedia.org/r/469566 (https://phabricator.wikimedia.org/T207841) [08:06:12] (03CR) 10Hashar: [C: 032] Record Junit files from Quibble PHPUnit run [integration/config] - 10https://gerrit.wikimedia.org/r/469566 (https://phabricator.wikimedia.org/T207841) (owner: 10Hashar) [08:06:58] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Quibble should instruct PHPUnit to generate Junit files - https://phabricator.wikimedia.org/T207841 (10hashar) I have deployed Quibble 0.0.28 and adjusted the jobs to add the Junit plugin for `log/junit*.xml` [08:10:47] (03Merged) 10jenkins-bot: Record Junit files from Quibble PHPUnit run [integration/config] - 10https://gerrit.wikimedia.org/r/469566 (https://phabricator.wikimedia.org/T207841) (owner: 10Hashar) [08:24:03] (03PS1) 10Hashar: Allow empty Junit files for Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/469570 (https://phabricator.wikimedia.org/T207841) [08:24:26] (03CR) 10Hashar: [C: 032] Allow empty Junit files for Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/469570 (https://phabricator.wikimedia.org/T207841) (owner: 10Hashar) [08:26:51] (03Merged) 10jenkins-bot: Allow empty Junit files for Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/469570 (https://phabricator.wikimedia.org/T207841) (owner: 10Hashar) [08:58:46] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10Discovery-Search (Current work), 10Patch-For-Review, 10Puppet: Elasticsearch puppet config changes broke puppet in various instances - https://phabricator.wikimedia.org/T205672 (10fgiunchedi) Patch merged, though ferm fails because of a known... [09:08:44] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10Nikerabbit) [09:21:47] hashar: cleaning up my gerrit dashboard, I see https://gerrit.wikimedia.org/r/c/integration/config/+/415001 that seems to be stalled for some time [09:21:53] how can I help? [09:32:40] (03PS1) 10Urbanecm: Add CI support for wikimedia-cz/events [integration/config] - 10https://gerrit.wikimedia.org/r/469589 (https://phabricator.wikimedia.org/T207879) [09:34:49] 10Beta-Cluster-Infrastructure, 10cloud-services-team: Access to deployment-redis3-changeprop broken - https://phabricator.wikimedia.org/T207825 (10jijiki) 05Open>03Invalid This is WIP, please refrain from using it. [09:35:09] (03CR) 10jerkins-bot: [V: 04-1] Add CI support for wikimedia-cz/events [integration/config] - 10https://gerrit.wikimedia.org/r/469589 (https://phabricator.wikimedia.org/T207879) (owner: 10Urbanecm) [09:43:19] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10Operations, 10SRE-Access-Requests: Add Lars Wirzenius to releng LDAP groups - https://phabricator.wikimedia.org/T207833 (10LarsWirzenius) @hashar @jijiki Thanks! I confirm that I can see logstash and grafana now. [12:08:24] 10Beta-Cluster-Infrastructure, 10cloud-services-team: Access to deployment-redis3-changeprop broken - https://phabricator.wikimedia.org/T207825 (10Krenair) 05Invalid>03Open Excuse me, but you left a broken instance in deployment-prep (if you want to do this please find another labs project). I (an unpaid v... [12:12:36] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10Discovery-Search (Current work), 10Patch-For-Review, 10Puppet: Elasticsearch puppet config changes broke puppet in various instances - https://phabricator.wikimedia.org/T205672 (10fgiunchedi) Looks like logs in deployment-prep are back now (cc... [12:21:51] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10Discovery-Search (Current work), 10Patch-For-Review, 10Puppet: Elasticsearch puppet config changes broke puppet in various instances - https://phabricator.wikimedia.org/T205672 (10dcausse) a:05dcausse>03Krenair I overlooked other instances... [12:28:48] 10Continuous-Integration-Config, 10Wikidata, 10wikidata-tech-focus, 10Patch-For-Review, 10User-Addshore: Move Wikibase to using the normal mediawiki extension (quibble) jobs - https://phabricator.wikimedia.org/T188717 (10hashar) The Quibble jobs now report again junit xml files (T207841) which offers a w... [12:29:36] gehel: re https://gerrit.wikimedia.org/r/#/c/integration/config/+/415001/ , I have no idea. I completely forgot the context about that change [12:30:40] gehel: it seems the site / site:stage goal is to generate online documentation, and in such case we do not care aobut running integration tests [12:30:49] (03PS3) 10Hashar: Skip integration tests in maven site publish job [integration/config] - 10https://gerrit.wikimedia.org/r/415001 [12:31:14] eventually I will want to reproduce the issue I had at the time and forgot to copy/paste [12:33:22] hashar: the maven site should contain test results, coverage, etc... so we do need to run them [12:33:58] hashar: if you go back to it and need some help, ping me, I'll remove myself from the change for now [12:37:20] gehel: so should we drop the -DskipTests as well ? [12:37:49] I guess I proposed that change to avoid running all tests, but if we need their test results to build the site ... [12:37:54] then all tests should be run :] [12:38:29] strangely, in the currently generated sites, I do see coverage [12:38:29] https://doc.wikimedia.org/wikidata-query-rdf/parent/common/jacoco/index.html [12:38:54] not sure how that's possible if we skip tests [12:39:42] coverage report on integration tests are useless, and all tests need to be green to merge, so test results are also mostly useless [12:40:54] maybe that patch is superseeded by something else [12:46:07] hmm [12:46:25] * hashar looks at https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-site-publish/ [12:47:26] ah no [12:47:28] that job is no more used [12:48:14] https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-java8-docker-site-publish/109/consoleFull [12:48:29] which uses clean install site site:stage [12:52:22] (03Abandoned) 10Hashar: Skip integration tests in maven site publish job [integration/config] - 10https://gerrit.wikimedia.org/r/415001 (owner: 10Hashar) [12:52:36] gehel: I am abandonning the change. We actually need the tests to be run or the site goal fails [12:53:00] ok [12:53:10] name: wikidata-query-rdf [12:53:14] # We need tests for org.wikidata.query.rdf.tool.Proxy - T190042 [12:53:15] maven_args: 'clean install site site:stage' [12:53:15] T190042: Migrate CI job wikidata-query-rdf-maven-site-publish to use a Docker container - https://phabricator.wikimedia.org/T190042 [12:56:38] (03PS1) 10Hashar: Clean useless maven overrides in publish jobs [integration/config] - 10https://gerrit.wikimedia.org/r/469610 [12:57:01] gehel: magic cleanup is https://gerrit.wikimedia.org/r/#/c/integration/config/+/469610 (which does not change anything in jenkins jobs). Merci! [12:59:09] hashar: I'll trust you on that one! [13:03:33] (03CR) 10Hashar: [C: 032] Clean useless maven overrides in publish jobs [integration/config] - 10https://gerrit.wikimedia.org/r/469610 (owner: 10Hashar) [13:08:04] (03Merged) 10jenkins-bot: Clean useless maven overrides in publish jobs [integration/config] - 10https://gerrit.wikimedia.org/r/469610 (owner: 10Hashar) [13:30:32] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Recommendation-API, 10Research, and 2 others: Create the recommendation api DB in Beta - https://phabricator.wikimedia.org/T207795 (10bmansurov) Thanks, @Krenair. Can you also share any documentation on how to connect to the database? [13:45:20] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Recommendation-API, 10Research, and 2 others: Create the recommendation api DB in Beta - https://phabricator.wikimedia.org/T207795 (10Krenair) Hi @bmansurov, sure, it's the standard unencrypted MySQL to port 3306 on the above hosts. It's not as r... [13:49:33] 10Release-Engineering-Team (Watching / External), 10Core Platform Team ( Code Health (TEC13)), 10Core Platform Team Kanban (Doing), 10Epic, 10User-notice: Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733 (10Anomie) [14:28:15] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Recommendation-API, 10Research, and 2 others: Create the recommendation api DB in Beta - https://phabricator.wikimedia.org/T207795 (10bmansurov) Thanks, @Krenair. This is very helpful. Where's the password stored? How can I get it? For the tools... [14:59:43] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Recommendation-API, 10Research, and 2 others: Create the recommendation api DB in Beta - https://phabricator.wikimedia.org/T207795 (10Krenair) >>! In T207795#4694636, @bmansurov wrote: > Where's the password stored? Well MySQL has the hash of co... [15:19:47] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Recommendation-API, 10Research, and 2 others: Create the recommendation api DB in Beta - https://phabricator.wikimedia.org/T207795 (10bmansurov) > You shouldn't need to directly interact with the password yourself, as I imagine puppet will just d... [15:20:10] Project beta-update-databases-eqiad build #29292: 04FAILURE in 9.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29292/ [15:23:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Recommendation-API, 10Research, and 2 others: Create the recommendation api DB in Beta - https://phabricator.wikimedia.org/T207795 (10Krenair) >>! In T207795#4694826, @bmansurov wrote: >> You shouldn't need to directly interact with the password... [15:24:49] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Recommendation-API, 10Research, and 2 others: Create the recommendation api DB in Beta - https://phabricator.wikimedia.org/T207795 (10bmansurov) > Just be careful not to show/store anything private to/on it as it's on a labs system. OK! [15:38:45] 10Project-Admins: Project creation request: GrowthExperiments - https://phabricator.wikimedia.org/T207907 (10Aklapper) I usually link to some wiki page in the project description for more info; is there something better than linking to currently non-existing https://www.mediawiki.org/wiki/Extension:GrowthExperim... [15:40:50] 10Continuous-Integration-Config, 10Wikidata, 10wikidata-tech-focus, 10Patch-For-Review, 10User-Addshore: Move Wikibase to using the normal mediawiki extension (quibble) jobs - https://phabricator.wikimedia.org/T188717 (10WMDE-leszek) Thanks @hashar for looking into this. A quick question checking that I... [16:07:11] (03CR) 10Thcipriani: [C: 032] "Redeployed service-pipeline* jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/469054 (owner: 10Thcipriani) [16:09:48] (03Merged) 10jenkins-bot: Ensure pipeline images are cleaned in production [integration/config] - 10https://gerrit.wikimedia.org/r/469054 (owner: 10Thcipriani) [16:20:10] Project beta-update-databases-eqiad build #29293: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29293/ [16:34:44] 10Beta-Cluster-Infrastructure, 10cloud-services-team: Access to deployment-redis3-changeprop broken - https://phabricator.wikimedia.org/T207825 (10jijiki) 05Open>03Resolved [17:09:01] Someone should voice liw :) [17:09:32] could do it easily but not permanently [17:10:07] looks like only greg could do it permanently [17:10:32] 10Continuous-Integration-Config, 10Wikidata, 10wikidata-tech-focus, 10Patch-For-Review, 10User-Addshore: Move Wikibase to using the normal mediawiki extension (quibble) jobs - https://phabricator.wikimedia.org/T188717 (10hashar) > The point of having "client" job is to have tests from group WikibaseClien... [17:12:23] 10Continuous-Integration-Config, 10Wikidata, 10wikidata-tech-focus, 10Patch-For-Review, 10User-Addshore: Move Wikibase to using the normal mediawiki extension (quibble) jobs - https://phabricator.wikimedia.org/T188717 (10Addshore) > And then just run the group WikibaseClient, which is doable in Quibble w... [17:20:14] Project beta-update-databases-eqiad build #29294: 04STILL FAILING in 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29294/ [17:21:21] !log (beta): Update mobileapps to 58cbdff [17:21:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:05:45] 10Project-Admins, 10Wikimedia-Technical-Conference-2018: Create "modularization" phabricator project - https://phabricator.wikimedia.org/T207976 (10bd808) [18:08:50] 10Project-Admins, 10Wikimedia-Technical-Conference-2018: Create "decoupling" phabricator project - https://phabricator.wikimedia.org/T207976 (10bd808) [18:09:17] 10Project-Admins, 10Wikimedia-Technical-Conference-2018: Create "decoupling" phabricator project - https://phabricator.wikimedia.org/T207976 (10bd808) Let's get some votes on the color of this bikeshed [18:10:42] bd808: "Conscious uncoupling"? [18:10:55] * James_F coughs. [18:20:04] Hi, this patch is not responding to "recheck": https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/ContentTranslation/+/469603/ anybody has an idea what's going on? [18:20:19] Project beta-update-databases-eqiad build #29295: 04STILL FAILING in 18 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29295/ [18:20:57] stephanebisson: it's in zuul [18:21:10] https://integration.wikimedia.org/zuul/ [18:21:41] currently waiting on https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/8743/ and https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/22225/ to finish [18:22:27] thcipriani: ah, ok. So it responded to my latest "recheck" at least. it ignored the one from ~5h ago. [18:22:57] well zeljkof misspelled "recheck" :) [18:23:06] > reckeck [18:23:26] ahah, and I missread [18:26:25] maybe we should match against [recheck]{7} ;) [18:26:49] * James_F laughs. [18:31:37] /r[ech]{3,5}k/ [18:32:05] cherek [18:32:41] nope, you clearly were conveying a different idea. [18:34:00] maybe just make a button? /me goes back away [18:34:16] greg-g: That's just crazy talk. [18:34:53] I know I know [19:12:01] thcipriani: oops :) [19:12:43] :P [19:20:15] Project beta-update-databases-eqiad build #29296: 04STILL FAILING in 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29296/ [19:28:26] 10Project-Admins: Project creation request: GrowthExperiments - https://phabricator.wikimedia.org/T207907 (10Catrope) I plan to create the extension page soon, but in the interim https://www.mediawiki.org/wiki/Growth/Personalized_first_day is probably the closest thing. [19:36:36] 10MediaWiki-Releasing, 10Growth-Team, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650 (10Nirmos) [19:46:33] 10Project-Admins: Project creation request: GrowthExperiments - https://phabricator.wikimedia.org/T207907 (10Catrope) Update: creating the extension page was easier than I thought, it's there now. [19:56:22] (03PS1) 10Hashar: docker: container for phpmetrics/phpmetrics v2.4.1 [integration/config] - 10https://gerrit.wikimedia.org/r/469689 (https://phabricator.wikimedia.org/T205133) [19:56:24] (03PS1) 10Hashar: Create mediawiki-core-phpmetrics-docker [integration/config] - 10https://gerrit.wikimedia.org/r/469690 (https://phabricator.wikimedia.org/T205133) [20:02:30] (03CR) 10Hashar: [C: 032] docker: container for phpmetrics/phpmetrics v2.4.1 [integration/config] - 10https://gerrit.wikimedia.org/r/469689 (https://phabricator.wikimedia.org/T205133) (owner: 10Hashar) [20:04:13] (03Merged) 10jenkins-bot: docker: container for phpmetrics/phpmetrics v2.4.1 [integration/config] - 10https://gerrit.wikimedia.org/r/469689 (https://phabricator.wikimedia.org/T205133) (owner: 10Hashar) [20:05:56] (03PS2) 10Hashar: Create mediawiki-core-phpmetrics-docker [integration/config] - 10https://gerrit.wikimedia.org/r/469690 (https://phabricator.wikimedia.org/T205133) [20:12:07] Hi, I have problems with git [20:12:24] zoran@zoran-notebook:~/development/mediawiki$ git fetch [20:12:25] Terminated [20:12:25] zoran@zoran-notebook:~/development/mediawiki$ git fetch && git pull [20:12:25] packet_write_wait: Connection to 208.80.154.85 port 29418: Broken pipe [20:12:25] fatal: internal server error [20:12:26] zoran@zoran-notebook:~/development/mediawiki$ [20:13:51] What I should do? [20:13:56] Is there any problem with gerrit? [20:14:08] Project mediawiki-core-phpmetrics-docker build #1: 04FAILURE in 1 min 33 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics-docker/1/ [20:15:58] With my connection is everything ok [20:16:22] well that's interesting [20:16:36] Zoranzoki21, are you able to ssh to gerrit.wikimedia.org port 29418? [20:17:05] I can SSH gerrit without problems [20:17:19] zoran@zoran-notebook:~$ ssh gerrit.wikimedia.org -l zoranzoki21 -p 29418 [20:17:20] **** Welcome to Gerrit Code Review **** [20:17:20] Hi Zoranzoki21, you have successfully connected over SSH. [20:17:20] Unfortunately, interactive shells are disabled. [20:17:20] To clone a hosted Git repository, use: [20:17:20] git clone ssh://zoranzoki21@gerrit.wikimedia.org:29418/REPOSITORY_NAME.git [20:17:24] Connection to gerrit.wikimedia.org closed. [20:17:26] zoran@zoran-notebook:~$ [20:18:31] I never had problems like this with git/gerrit [20:18:40] fatal: internal server error [20:18:40] This is first time [20:18:47] i would be interested in what " fatal: internal server error" [20:18:48] is [20:19:00] any of releng be able to look into the logs please ^^? [20:19:12] Yippee, build fixed! [20:19:13] Project mediawiki-core-phpmetrics-docker build #2: 09FIXED in 1 min 40 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics-docker/2/ [20:20:14] twentyafterfour or thcipriani ^^ [20:20:17] Project beta-update-databases-eqiad build #29297: 04STILL FAILING in 17 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29297/ [20:20:55] paladox: Let's talk about my problem here. No on two-three channels [20:20:59] ok [20:21:06] So I can easily watch [20:21:12] paladox: I'm dealing with the train right now but I can try to look at logs in a bit [20:21:15] I will try now from terminal in vcs [20:21:19] thanks [20:22:09] Can you check connections to gerrit? [20:22:18] Maybe gerrit have much connections correct [20:22:22] *currently [20:23:48] git fetch and git pull work for me [20:23:58] though i did just clone it fresh [20:24:41] (03PS3) 10Hashar: Create mediawiki-core-phpmetrics-docker [integration/config] - 10https://gerrit.wikimedia.org/r/469690 (https://phabricator.wikimedia.org/T205133) [20:27:39] 10Continuous-Integration-Config, 10Advanced-Search, 10TCB-Team, 10Wikibase-Quality-Constraints, 10Wikidata: Update grunt to 1.0.3 for AdvancedSearch and WikibaseQualityConstraints - https://phabricator.wikimedia.org/T207988 (10Umherirrender) [20:27:53] (03CR) 10jerkins-bot: [V: 04-1] Create mediawiki-core-phpmetrics-docker [integration/config] - 10https://gerrit.wikimedia.org/r/469690 (https://phabricator.wikimedia.org/T205133) (owner: 10Hashar) [20:27:58] git works for me as well [20:28:04] I can't reproduce [20:28:14] (03CR) 10Hashar: "deployed and creates https://doc.wikimedia.org/mediawiki-core/master/phpmetrics/" [integration/config] - 10https://gerrit.wikimedia.org/r/469690 (https://phabricator.wikimedia.org/T205133) (owner: 10Hashar) [20:28:21] (03CR) 10Hashar: [C: 032] Create mediawiki-core-phpmetrics-docker [integration/config] - 10https://gerrit.wikimedia.org/r/469690 (https://phabricator.wikimedia.org/T205133) (owner: 10Hashar) [20:29:14] I am back [20:29:24] I called support of my provider [20:31:32] And they told me to it is problem with server where is placed gerrit [20:31:37] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10mmodell) [20:32:28] there coulden't be any problem with gerrit [20:32:36] or the server [20:32:48] since me and twentyafterfour git successfully [20:33:01] (03Merged) 10jenkins-bot: Create mediawiki-core-phpmetrics-docker [integration/config] - 10https://gerrit.wikimedia.org/r/469690 (https://phabricator.wikimedia.org/T205133) (owner: 10Hashar) [20:33:22] What I should check [20:33:55] SSH keys are correct (same on laptop on gerrit on wikitech) [20:34:02] My connection is ok [20:34:33] yes they are correct [20:34:40] otherwise you would be getting a auth error [20:34:46] I restarted laptop [20:34:56] packet_write_wait: Connection to 208.80.154.85 port 29418: Broken pipe [20:34:57] Now I got: [20:34:58] zoran@zoran-notebook:~/development/mediawiki$ git fetch && git pull [20:34:58] fatal: internal server error [20:34:58] fatal: protocol error: bad line length character: eERR [20:35:02] that indicates a network problem [20:35:11] I restarted router [20:35:15] sounds network related [20:35:35] Network is ok (I and support from Telenor checked) [20:35:43] So is gerrit [20:35:47] since it works for me and twentyafterfour [20:36:05] I will try to clone another repository [20:36:09] on example [20:36:49] it will likley fail if some where in your network is interrupting ssh [20:37:32] Cloning works without problems [20:38:38] I cloned repository without problems [20:38:59] So, we know now.. Cloning works without problems. [20:39:03] git pull no works [20:39:06] git fetch no works [20:39:11] git pull && git fetch no works [20:39:14] Other operations works [20:39:47] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10mmodell) [20:41:41] Gerrit is very slow for me now [20:44:16] can you traceroute to gerrit? [20:44:34] Sure [20:44:37] I done speedtest again [20:45:05] zoran@zoran-notebook:~/development/mediawiki-config$ speedtest-cli [20:45:05] Retrieving speedtest.net configuration... [20:45:05] Testing from Telenor d.o.o. Beograd (109.245.127.73)... [20:45:05] Retrieving speedtest.net server list... [20:45:05] Selecting best server based on ping... [20:45:06] Hosted by BeotelNet (Belgrade) [0.52 km]: 6.004 ms [20:45:08] Testing download speed................................................................................ [20:45:11] Download: 20.00 Mbit/s [20:45:13] Testing upload speed...................................................................................................... [20:45:16] Upload: 200.45 Mbit/s [20:45:20] zoran@zoran-notebook:~/development/mediawiki-config$ [20:45:22] I will traceroute gerrit now [20:45:24] Should I traceroute domain or IP? [20:45:25] I see a log of "IOException: Connection reset by peer" [20:45:31] either [20:45:34] in gerrit logs as part of git upload-pack [20:45:41] I don't think the problem is the DNS resolution [20:46:53] Ok [20:47:00] Can I traceroute domain or IP? [20:47:06] lots of timeouts as well [20:47:37] Use a paatebin in the future please [20:47:39] Pastebin [20:47:44] > java.net.SocketTimeoutException: timeout exceeded: 30000 [20:47:49] ok [20:47:58] I will traceroute IP and domain [20:48:44] logs make it sound like your connection is a bit slow and it's trying to download a lot of stuff. [20:49:47] try using the http remote rather than the ssh remote, perhaps [20:50:11] Why is ping so big to servers of WMF? [20:50:36] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10Krinkle) [20:50:38] Could be routing issues [20:50:52] Also eqiad is in the US [20:51:02] so if you are far away from it ping will show that [20:51:19] from the UK it's 87ms [20:51:30] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10mmodell) [20:51:36] I am in Serbia [20:51:36] Zoranzoki21, how big are we talking? [20:51:44] 170 164 150 [20:51:48] ms [20:52:30] you should be ok [20:53:40] Check logs now [20:53:54] I trying to pull mediawiki-config [20:54:46] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10mmodell) So we're now at group1, the error rate seems to have stabilized. I want to monitor it just a few more minutes before I decide for... [20:55:21] Which option in git command I can use for debugging (verbose output)? [20:55:33] I want to try it [20:55:41] Maybe it can help (if git have it option) [20:56:06] could you try to clone from https://gerrit.wikimedia.org/r/operations/mediawiki-config please? [20:56:15] Cloning works without problems [20:56:59] you said you're trying to pull? What command are you running? [20:57:04] git pull [20:57:12] I trying to pull [20:57:21] For gerrit.wikimedia.org no works. For github.com works without problems [20:59:52] git remote add gerrit-https https://gerrit.wikimedia.org/r/operations/mediawiki-config && git pull gerrit-https [20:59:56] might work [21:00:06] different timeout [21:00:13] same source [21:00:34] the github mirror should also be up-to-date FWIW [21:01:45] no works [21:03:46] Same behaviour [21:04:22] can you paste what you're seeing in a pastebin? [21:05:22] Only I got: Terminated [21:05:39] When I pasted what you told me [21:05:49] But, after three minutes [21:09:04] Terminated? [21:09:14] I'm not sure this is a network problem [21:09:19] yes [21:09:24] I get ''Terminated'' [21:10:52] 10Release-Engineering-Team (Watching / External), 10Core Platform Team ( Code Health (TEC13)), 10Core Platform Team Kanban (Doing), 10Epic, 10User-notice: Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733 (10MusikAnimal) I understand backfilling comments is still on the to-dos... [21:11:47] Can you restart gerrit? [21:12:53] I'm not sure that will fix matters in this instance. [21:13:48] I'm not really sure what can resolve this problem [21:14:44] thcipriani that will most deftly not fix his problem [21:14:51] since it works for us [21:15:01] (no other users are reporting that problem) [21:15:17] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10mmodell) at group1 and 66% of errors are from `1.33.0-wmf.1`. Normally it should be more like 10%-20% of errors from the group1 branch. [21:15:25] and secondly "packet_write_wait: Connection to 208.80.154.85 port 29418: Broken pipe" [21:15:37] is when ssh looses network access or something else. [21:16:15] See this [21:16:54] https://pastebin.com/nDqwZAA9 [21:16:58] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10mmodell) Going ahead with group2 and will see how it goes. [21:17:31] ssh has a timeout of 30 seconds [21:18:10] omg [21:18:18] ? [21:19:11] ... [21:19:34] you wrote oh my god so what supprises you? [21:19:39] did you discover something? [21:19:48] ssh has a timeout of 30 seconds [21:19:56] yep [21:20:01] that should work [21:20:10] the 30 second timeout is if it dosen't do anything [21:20:13] just sits there [21:20:13] Project beta-update-databases-eqiad build #29298: 04STILL FAILING in 13 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29298/ [21:20:21] try https as I mentioned [21:20:27] I tryed with https [21:20:29] ^^ [21:20:29] Same behaviour [21:20:33] I tryed with https [21:20:35] Same behaviour [21:20:44] Zoranzoki21 https://stackoverflow.com/questions/6178401/how-can-i-debug-git-git-shell-related-problems [21:21:28] if github works: git clone https://github.com/wikimedia/operations-mediawiki-config && cd operations-mediawiki-config && git remote origin set-url ssh://zoranzoki21@gerrit.wikimedia.org:29418/operations/mediawiki-config [21:22:12] I will put output when it finish. Then, I going to sleep. if this no works tomorrow, I will open task on phabricator [21:22:24] sounds good [21:22:30] oran@zoran-notebook:~/development/mediawiki$ GIT_CURL_VERBOSE=1 GIT_TRACE=1 git pull origin master [21:22:30] 23:21:16.569179 git.c:344 trace: built-in: git pull origin master [21:22:30] 23:21:16.643166 run-command.c:640 trace: run_command: git fetch --update-head-ok origin master [21:22:30] 23:21:16.645421 git.c:344 trace: built-in: git fetch --update-head-ok origin master [21:22:31] 23:21:18.279164 run-command.c:640 trace: run_command: unset GIT_DIR GIT_PREFIX; ssh -p 29418 zoranzoki21@gerrit.wikimedia.org 'git-upload-pack '\''/mediawiki/core'\''' [21:22:50] use a pastebin [21:23:06] please [21:23:10] Ok ok. I will [21:23:42] Nothing no happening [21:24:38] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10mmodell) [21:25:01] If this continue to happening tomorrow, I will open task on Phabricator [21:25:04] I going to sleep now [21:25:07] Good night! [21:33:54] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10mmodell) [21:58:34] 10Release-Engineering-Team (Watching / External), 10Core Platform Team ( Code Health (TEC13)), 10Core Platform Team Kanban (Doing), 10Epic, 10User-notice: Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733 (10Anomie) We could start backfilling as soon as Monday, although I have... [22:20:11] Project beta-update-databases-eqiad build #29299: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29299/ [22:53:00] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T206655 (10mmodell) [23:20:10] Project beta-update-databases-eqiad build #29300: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29300/