[00:02:22] 10Phabricator, 10Project-Admins: Add links on all Phabricator/Maniphest project pages to the corresponding source code pages in Diffusion - https://phabricator.wikimedia.org/T144127 (10Huji) For the record, I have been going around and adding such links whenever I run into a project in Phab that doesn't have a... [00:51:27] (03CR) 10Xqt: [C: 03+1] "I’ve submitted the other patch" [integration/config] - 10https://gerrit.wikimedia.org/r/495815 (owner: 10Hashar) [01:03:04] 10Phabricator: Remove/Reduce large bug report header for "Edit" form - https://phabricator.wikimedia.org/T207525 (10mmodell) 05Open→03Resolved a:03mmodell [01:18:20] 10Phabricator, 10Repository-Admins: Remove mentioning rPHES in description of rPHAB - https://phabricator.wikimedia.org/T217795 (10mmodell) 05Open→03Resolved a:03mmodell Updated the description. [07:10:13] maintenance-disconnect-full-disks build 54071 integration-slave-jessie-1001 (/srv: 95%): OFFLINE due to disk space [07:27:02] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Is this related to T202470? If so, should we mention the ticket number in the commit message?" [integration/config] - 10https://gerrit.wikimedia.org/r/495641 (owner: 10Legoktm) [07:28:09] (03CR) 10Thiemo Kreuz (WMDE): "Patch set 3 is empty after the latest rebase. Seems what this patch aimed to do was already done in another patch. Abandon?" [integration/config] - 10https://gerrit.wikimedia.org/r/463912 (owner: 10Legoktm) [07:30:12] maintenance-disconnect-full-disks build 54075 integration-slave-jessie-1001: OFFLINE due to disk space [07:37:18] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Upstream: chromium 72 crash when used with --remote-debugging-port - https://phabricator.wikimedia.org/T216702 (10hashar) 05Open→03Resolved a:03hashar Debian has released a new version of Chromium 72.0.3626.122-1~deb9u1 and it works fine.... [07:38:44] (03CR) 10Hashar: [C: 03+2] "We should have done the CI change first ;) Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/495815 (owner: 10Hashar) [07:41:02] (03Merged) 10jenkins-bot: Rename pywikibot envs [integration/config] - 10https://gerrit.wikimedia.org/r/495815 (owner: 10Hashar) [07:55:13] maintenance-disconnect-full-disks build 54080 integration-slave-jessie-1001: OFFLINE due to disk space [07:59:38] 10Continuous-Integration-Infrastructure, 10Operations, 10Traffic, 10Patch-For-Review: Make CI run Varnish VCL tests - https://phabricator.wikimedia.org/T128188 (10ema) 05Resolved→03Open >>! In T128188#4841316, @hashar wrote: > I am pretty sure @ema finished up the integration of varnishtest with CI / r... [08:20:12] maintenance-disconnect-full-disks build 54085 integration-slave-jessie-1001: OFFLINE due to disk space [08:45:29] maintenance-disconnect-full-disks build 54090 integration-slave-jessie-1001: OFFLINE due to disk space [09:10:13] maintenance-disconnect-full-disks build 54095 integration-slave-jessie-1001: OFFLINE due to disk space [09:27:26] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10zeljkofilipin) a:03zeljkofilipin [09:27:39] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10zeljkofilipin) [09:35:13] maintenance-disconnect-full-disks build 54100 integration-slave-jessie-1001: OFFLINE due to disk space [09:35:28] 10Release-Engineering-Team, 10MediaWiki-Database, 10MediaWiki-extensions-CodeReview: CodeRevisionListView::getRevCount is creating slow queries on mediawiki.org - https://phabricator.wikimedia.org/T218079 (10jcrespo) [09:48:46] 10Release-Engineering-Team, 10MediaWiki-Database, 10MediaWiki-extensions-CodeReview: CodeRevisionListView::getRevCount is creating slow queries on mediawiki.org - https://phabricator.wikimedia.org/T218079 (10jcrespo) [09:52:27] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10MediaWiki-extensions-CodeReview: CodeReview extension: Code stewardship review - https://phabricator.wikimedia.org/T205482 (10jcrespo) Causing issues with bad performant queries on mediawiki.org (s3) databases: T218079 [09:53:16] 10Release-Engineering-Team, 10MediaWiki-Database, 10MediaWiki-extensions-CodeReview: CodeRevisionListView::getRevCount is creating slow queries on mediawiki.org - https://phabricator.wikimedia.org/T218079 (10jcrespo) [10:00:13] maintenance-disconnect-full-disks build 54105 integration-slave-jessie-1001: OFFLINE due to disk space [10:07:09] 10Release-Engineering-Team, 10MediaWiki-Database, 10MediaWiki-extensions-CodeReview: CodeRevisionListView::getRevCount is creating slow queries on mediawiki.org - https://phabricator.wikimedia.org/T218079 (10hashar) That SQL query has been in the code and deployed since at least 2011. The reason is most def... [10:22:25] 10Beta-Cluster-Infrastructure, 10Discovery-Search, 10MediaWiki-Search, 10Services (next): Beta Cluster search box displays unexisting pages as results - https://phabricator.wikimedia.org/T186993 (10Cparle) Testing Structured Data on Commons on beta is blocked by this ... @Pchelolo is this something you're... [10:25:10] 10Release-Engineering-Team, 10MediaWiki-Database, 10MediaWiki-extensions-CodeReview: CodeRevisionListView::getRevCount is creating slow queries on mediawiki.org - https://phabricator.wikimedia.org/T218079 (10hashar) Going through EXPLAIN shows the query is rather straightforward using indices and going throu... [10:25:12] maintenance-disconnect-full-disks build 54110 integration-slave-jessie-1001: OFFLINE due to disk space [10:32:37] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10zeljkofilipin) [[ https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Create_the_new_branch_in_Gerrit | Cutting... [10:35:00] 10Beta-Cluster-Infrastructure, 10SDC General, 10Wikidata, 10Services (next): No jobs running on beta cluster - https://phabricator.wikimedia.org/T215339 (10Cparle) [10:50:20] maintenance-disconnect-full-disks build 54115 integration-slave-jessie-1001: OFFLINE due to disk space [10:53:18] (03PS3) 10Effie Mouzeli: Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) [10:55:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10hashar) Sorry I was wrong in my previous comment mentionin... [10:57:55] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689 (10hashar) It is a... [10:58:27] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [10:58:29] (03CR) 10jerkins-bot: [V: 04-1] Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [11:00:46] (03PS4) 10Effie Mouzeli: Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) [11:02:19] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [11:02:21] (03CR) 10jerkins-bot: [V: 04-1] Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [11:14:17] 10Beta-Cluster-Infrastructure, 10SDC General, 10Wikidata, 10Services (next): No jobs running on beta cluster - https://phabricator.wikimedia.org/T215339 (10Krenair) can we perhaps do a bit of iptables magic to redirect traffic to the right port without having to mess around with puppet, templates etc.? [11:15:12] maintenance-disconnect-full-disks build 54120 integration-slave-jessie-1001: OFFLINE due to disk space [11:40:12] maintenance-disconnect-full-disks build 54125 integration-slave-jessie-1001: OFFLINE due to disk space [11:53:35] 10Continuous-Integration-Infrastructure, 10Operations, 10Packaging, 10Patch-For-Review: Upgrade jenkins-debian-glue to v0.20.0 - https://phabricator.wikimedia.org/T212774 (10jbond) @hashar i have taken another look at the patch i created yesterday and i now think it is incorrect. As far as i can tell `pr... [12:05:13] maintenance-disconnect-full-disks build 54130 integration-slave-jessie-1001: OFFLINE due to disk space [12:30:13] maintenance-disconnect-full-disks build 54135 integration-slave-jessie-1001: OFFLINE due to disk space [12:53:27] (03CR) 10Thcipriani: "posted some drive-by comments that are hopefully helpful" (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/495160 (owner: 10MarkAHershberger) [12:55:12] maintenance-disconnect-full-disks build 54140 integration-slave-jessie-1001: OFFLINE due to disk space [13:00:58] Project beta-scap-eqiad build #241130: 04FAILURE in 3.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241130/ [13:14:34] Yippee, build fixed! [13:14:34] Project beta-scap-eqiad build #241131: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241131/ [13:20:12] maintenance-disconnect-full-disks build 54145 integration-slave-jessie-1001: OFFLINE due to disk space [13:31:44] Thats ironic ^ [13:33:56] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10zeljkofilipin) 1.33.0-wmf.21 at group 0 ([[ https://tools.wmflabs.org/sal/log/AWlyE0lRIm9Dp5A3YxC4 | sal ]], [[ https://gerrit.wikimedi... [13:43:33] (03PS3) 10Thcipriani: Add Srishti Sethi to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/493139 (owner: 10BryanDavis) [13:45:13] maintenance-disconnect-full-disks build 54150 integration-slave-jessie-1001: OFFLINE due to disk space [13:46:21] (03CR) 10Thcipriani: [C: 03+2] Add Srishti Sethi to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/493139 (owner: 10BryanDavis) [13:47:46] (03Merged) 10jenkins-bot: Add Srishti Sethi to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/493139 (owner: 10BryanDavis) [13:49:49] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10zeljkofilipin) [13:50:10] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/493139/ [13:50:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:22:26] 10Scap: Newline in scap sync-file message: no notification, no SAL entry - https://phabricator.wikimedia.org/T217558 (10thcipriani) p:05Triage→03Normal I do see the message in its entirety in logstash Synchronized wmf-config/InitialiseSettings.php: T209857 Increase CPU benchmark sampling factor (duration:... [14:35:15] maintenance-disconnect-full-disks build 54160 integration-slave-jessie-1001: OFFLINE due to disk space [15:00:16] maintenance-disconnect-full-disks build 54165 integration-slave-jessie-1001: OFFLINE due to disk space [15:25:13] maintenance-disconnect-full-disks build 54170 integration-slave-jessie-1001: OFFLINE due to disk space [15:42:35] 10Phabricator, 10Reading-Infrastructure-Team-Backlog: Update Herald (H228) to include project #wikimediaeditortasks (3907) - https://phabricator.wikimedia.org/T218114 (10Jhernandez) [15:50:16] maintenance-disconnect-full-disks build 54175 integration-slave-jessie-1001: OFFLINE due to disk space [15:52:59] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:54:26] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [3.0] [15:54:39] 10Phabricator, 10Reading-Infrastructure-Team-Backlog: Update Herald (H228) to include project #wikimediaeditortasks (3907) - https://phabricator.wikimedia.org/T218114 (10MBinder_WMF) 05Open→03Resolved {meme, src="seal-of-approval"} [16:15:14] maintenance-disconnect-full-disks build 54180 integration-slave-jessie-1001: OFFLINE due to disk space [16:16:03] hi, is there something I need to do if I want an MW extension to appear on https://doc.wikimedia.org/? [16:16:54] I don't particularly want to be linked from the main page but a link where the doc is generated [16:19:28] (03PS3) 10Hashar: java8: set MAVEN_USER_HOME to a writeable directory [integration/config] - 10https://gerrit.wikimedia.org/r/495887 (https://phabricator.wikimedia.org/T218099) (owner: 10Gehel) [16:19:55] (03CR) 10Hashar: [C: 03+2] "Sorry it took a while. I cant no update the Jenkins jobs myself though but I will rebuild the images." [integration/config] - 10https://gerrit.wikimedia.org/r/495887 (https://phabricator.wikimedia.org/T218099) (owner: 10Gehel) [16:22:00] (03Merged) 10jenkins-bot: java8: set MAVEN_USER_HOME to a writeable directory [integration/config] - 10https://gerrit.wikimedia.org/r/495887 (https://phabricator.wikimedia.org/T218099) (owner: 10Gehel) [16:28:04] PROBLEM - Host integration-publishing02 is DOWN: CRITICAL - Host Unreachable (172.16.4.5) [16:28:17] dcausse: On the main page, it's a static html file in the integration/docroot repo [16:28:56] Reedy: thanks, but to generate the doc itself do I need to configure something special in CI? [16:29:29] Oh, you mean you want CI to do the doxygen (or whatever for JS) generation? [16:29:47] I want PHP doc but yes [16:30:02] I tried to guess the URL but looks like it's not generated [16:30:23] I'm not sure if we support phpdoc on this stuff [16:30:50] yes we do https://doc.wikimedia.org/mediawiki-core/master/php/ [16:30:57] That's doxygen [16:30:59] See the bottom right [16:31:18] so we are talking about the same thing :P [16:31:19] oh then yes I want doxygen :) [16:31:48] Which extension is it? [16:31:52] CirrusSearch [16:32:02] I think, in zuul/layout.yaml you just need to make sure - mwext-doxygen-publish is a postmerge job [16:32:22] Reedy: great, thanks! I'll take a look [16:32:24] https://github.com/wikimedia/integration-config/blob/master/zuul/layout.yaml#L3705-L3706 [16:32:32] Like is done there for CollaborationKit [16:32:49] So should just be a copy paste to just after https://github.com/wikimedia/integration-config/blob/master/zuul/layout.yaml#L3647-L3652 [16:33:04] perfect, will do that now [16:33:33] Do you have deploy access for CI? If not, I should be able to push it out for you [16:34:01] no I don't so yes I might need your help again :) [16:34:10] You might need/want to add a Doxyfile to CirrusSearch (if there isn't one already) [16:34:22] (there isn't one in my checkouts from yesterday) [16:35:51] ah [16:36:00] But we can do that after if necessary [16:36:25] Can probably just steal what CollaborationKit (or similar) has anyway [16:36:42] will do [16:36:47] (03PS1) 10DCausse: Add doxygen for CirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/495931 [16:37:48] (03CR) 10Reedy: [C: 04-1] Add doxygen for CirrusSearch (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/495931 (owner: 10DCausse) [16:40:08] it was an easy patch and I got it wrong :P [16:40:19] maintenance-disconnect-full-disks build 54185 integration-slave-jessie-1001: OFFLINE due to disk space [16:40:36] :D [16:40:56] (03PS2) 10DCausse: Add doxygen for CirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/495931 [16:41:09] (03CR) 10DCausse: Add doxygen for CirrusSearch (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/495931 (owner: 10DCausse) [16:41:40] I have also this: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CirrusSearch/+/495932 [16:41:44] I copied from MobileFrontend [16:42:19] I imagine it'll be at least 99% right for what you want [16:42:29] (03CR) 10Reedy: [C: 03+2] Add doxygen for CirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/495931 (owner: 10DCausse) [16:44:40] (03Merged) 10jenkins-bot: Add doxygen for CirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/495931 (owner: 10DCausse) [16:45:26] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/495931 [16:45:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:48:46] (03PS1) 10Reedy: Add CirrusSearch doxygen [integration/docroot] - 10https://gerrit.wikimedia.org/r/495935 [16:49:48] (03PS2) 10Reedy: Add CirrusSearch doxygen [integration/docroot] - 10https://gerrit.wikimedia.org/r/495935 [16:50:13] dcausse: ^ we can merge that a bit later when the job has run [16:50:22] Reedy: ok [16:50:35] Exception in thread "main" java.io.FileNotFoundException: /nonexistent/.m2/wrapper/dists/apache-maven-3.6.0-bin/7q9b549jss6tgtr7gdokcthm4f/apache-maven-3.6.0-bin.zip.part (No such file or directory) [16:50:38] looks like the CI merge queue is a bit backlogged [16:50:56] anybody knows why maven builds are suddenly broken ^^? [16:51:21] (03CR) 10Dduvall: [C: 04-1] "Sorry for the delay! I was on vacation in Hawaii and not thinking about how to get NPM packages installed or really anything other than wh" [blubber] - 10https://gerrit.wikimedia.org/r/492922 (https://phabricator.wikimedia.org/T205911) (owner: 10Alexandros Kosiaris) [16:56:52] (03PS1) 10MacFan4000: Archive EducationProgram [integration/config] - 10https://gerrit.wikimedia.org/r/495939 (https://phabricator.wikimedia.org/T214457) [16:59:10] (03PS2) 10MacFan4000: Archive EducationProgram [integration/config] - 10https://gerrit.wikimedia.org/r/495939 (https://phabricator.wikimedia.org/T214457) [17:00:55] Project beta-scap-eqiad build #241151: 04FAILURE in 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241151/ [17:05:13] maintenance-disconnect-full-disks build 54190 integration-slave-jessie-1001: OFFLINE due to disk space [17:15:41] Yippee, build fixed! [17:15:42] Project beta-scap-eqiad build #241152: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241152/ [17:20:38] Project beta-update-databases-eqiad build #32372: 04FAILURE in 37 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32372/ [17:30:14] maintenance-disconnect-full-disks build 54195 integration-slave-jessie-1001: OFFLINE due to disk space [17:50:09] thcipriani: any chance you'd be around to help me on those maven builds? I know what the problem is, but not sure what the solution is [17:51:40] or anyone else who knows something about docker and our CI ? [17:51:56] for context, the breaking change is https://gerrit.wikimedia.org/r/c/integration/config/+/495887 [17:52:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10Krinkle) This does not belong in DevelopmentSettings.php.... [17:52:40] the MAVEN_USER_HOME is set as an env variable in https://gerrit.wikimedia.org/r/c/integration/config/+/495887/3/dockerfiles/java8/Dockerfile.template but isn't picked up by mvnw for some reason [17:52:49] * thcipriani looks [17:53:04] if I run docker locally and pass that variable through --env file, then it works as expected [17:53:42] we're doing something similar with Sonar, and it works in that case [17:55:12] maintenance-disconnect-full-disks build 54200 integration-slave-jessie-1001: OFFLINE due to disk space [17:55:38] thcipriani: thanks! [17:55:44] gehel: so I can say that the new java8 image at least has the variable set in the environment [17:56:01] and it's set to /cache/maven [17:56:42] gehel: do you have a broken build example run taht I can fiddle with? [17:57:05] thcipriani: https://integration.wikimedia.org/ci/job/search-extra-maven-java8-docker/147/console [17:58:07] thcipriani: or for an open change : https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-java8-docker/759/console [18:00:10] gehel: hrm, the first change is using docker-registry.wikimedia.org/releng/java8:0.5.0 vs the update that made 0.5.1 [18:00:39] 10Release-Engineering-Team (Kanban): Investigate Zuul v3 - https://phabricator.wikimedia.org/T218138 (10brennen) p:05Triage→03Normal [18:01:35] thcipriani: Oh, that first build was probably before I tried to fix the issue by adding this MAVEN_USER_HOME var [18:02:43] Oh, but as far as I can see, that build is using 0.3.0 and not 0.3.1 [18:03:15] that might actually be the issue [18:03:15] FWIW the 2nd build used 0.3.0 which doesn't have MAVEN_USER_HOME set either [18:03:41] ok, so what is missing to use 0.3.1? [18:03:49] 10Release-Engineering-Team (Kanban): Investigate Zuul v3 - https://phabricator.wikimedia.org/T218138 (10Paladox) Related to T186426 [18:04:08] (03PS6) 10Krinkle: castor: add --delay-updates to rsync commands [integration/config] - 10https://gerrit.wikimedia.org/r/479558 (https://phabricator.wikimedia.org/T203506) (owner: 10Thcipriani) [18:04:42] Oh, I think I found it [18:04:43] gehel: https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master/jjb/wikidata.yaml#24 [18:04:58] we'll need to update image tags in jjb and update jobs [18:05:26] ok, patch coming up! [18:05:55] 10Release-Engineering-Team (Kanban): Check gerrithub's info in spreadsheet - https://phabricator.wikimedia.org/T217890 (10brennen) Yeah, it doesn't really seem like this fits under CI tooling. Seems to be a Gerrit-to-GitHub integration of sorts, run by GerritForge, who do hosted Gerrit. Updated spreadsheet acc... [18:07:08] 10Release-Engineering-Team (Kanban): Evaluate sourcehut builds - https://phabricator.wikimedia.org/T217852 (10brennen) [18:07:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-production-error (Shared Build Failure): Workspaces for mwgate-php55lint / mwgate-php70lint are getting huge - https://phabricator.wikimedia.org/T179963 (10Krinkle) p:05High→03Normal Moving out of active list of iss... [18:09:07] 10Release-Engineering-Team (Kanban): Evaluate Zuul v3 - https://phabricator.wikimedia.org/T218138 (10brennen) [18:11:21] (03PS1) 10Gehel: java8: actually use the new image fixing the mvn wrapper cache issue. [integration/config] - 10https://gerrit.wikimedia.org/r/495955 (https://phabricator.wikimedia.org/T218099) [18:11:40] * thcipriani reviews [18:11:42] thcipriani: ^ I think this should fix the issue [18:11:47] thcipriani: thanks! [18:12:30] 10Release-Engineering-Team (Kanban), 10Zuul: Evaluate Zuul v3 - https://phabricator.wikimedia.org/T218138 (10brennen) [18:13:29] (03CR) 10Thcipriani: [C: 03+1] java8: actually use the new image fixing the mvn wrapper cache issue. (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/495955 (https://phabricator.wikimedia.org/T218099) (owner: 10Gehel) [18:13:41] gehel: looks like you missed xgboost [18:14:26] (03PS2) 10Gehel: java8: actually use the new image fixing the mvn wrapper cache issue. [integration/config] - 10https://gerrit.wikimedia.org/r/495955 (https://phabricator.wikimedia.org/T218099) [18:14:28] right! [18:14:47] xgboost should not be impacted, but still [18:16:32] (03CR) 10Gehel: java8: actually use the new image fixing the mvn wrapper cache issue. (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/495955 (https://phabricator.wikimedia.org/T218099) (owner: 10Gehel) [18:16:45] (03CR) 10Thcipriani: [C: 03+2] java8: actually use the new image fixing the mvn wrapper cache issue. [integration/config] - 10https://gerrit.wikimedia.org/r/495955 (https://phabricator.wikimedia.org/T218099) (owner: 10Gehel) [18:16:52] gehel: cool, I'll deploy [18:17:12] thcipriani: thanks a lot! Ping me when done and I'll recheck one of the project [18:17:35] will do, thanks for the patch (and knowing the problem :)) [18:17:49] Easy, I'm the one who created the problem :) [18:18:59] (03Merged) 10jenkins-bot: java8: actually use the new image fixing the mvn wrapper cache issue. [integration/config] - 10https://gerrit.wikimedia.org/r/495955 (https://phabricator.wikimedia.org/T218099) (owner: 10Gehel) [18:20:07] gehel: alright, all jobs should be updated. [18:20:13] maintenance-disconnect-full-disks build 54205 integration-slave-jessie-1001: OFFLINE due to disk space [18:20:26] thcipriani: re-checking [18:21:10] Yippee, build fixed! [18:21:10] Project beta-update-databases-eqiad build #32373: 09FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32373/ [18:21:31] thcipriani: looks like it is running fine now. Thanks a lot! [18:21:48] gehel: awesome, glad to hear all is well :) [18:23:33] !log bring integration-slave-jessie-1001 back online, /srv disk space now at 20% (not sure if someone cleared disk and forgot to repool) [18:23:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:24:41] (03CR) 10Reedy: [C: 03+2] Add CirrusSearch doxygen [integration/docroot] - 10https://gerrit.wikimedia.org/r/495935 (owner: 10Reedy) [18:25:16] (03Merged) 10jenkins-bot: Add CirrusSearch doxygen [integration/docroot] - 10https://gerrit.wikimedia.org/r/495935 (owner: 10Reedy) [18:25:22] (03CR) 10jenkins-bot: Add CirrusSearch doxygen [integration/docroot] - 10https://gerrit.wikimedia.org/r/495935 (owner: 10Reedy) [18:27:57] (03CR) 10Krinkle: [C: 03+2] "Trying to reduce npm failure rate." [integration/config] - 10https://gerrit.wikimedia.org/r/479558 (https://phabricator.wikimedia.org/T203506) (owner: 10Thcipriani) [18:30:07] (03Merged) 10jenkins-bot: castor: add --delay-updates to rsync commands [integration/config] - 10https://gerrit.wikimedia.org/r/479558 (https://phabricator.wikimedia.org/T203506) (owner: 10Thcipriani) [18:38:04] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/479558 / T203506 [18:38:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:38:06] T203506: Jenkins jobs for MediaWiki failing with 'npm: shasum check failed' - https://phabricator.wikimedia.org/T203506 [18:38:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Reading-Infrastructure-Team-Backlog, 10Wikimedia-production-error (Shared Build Failure): Jenkins jobs for MediaWiki failing with 'npm: shasum check failed' - https://phabricator.wikimedia.org/T203506 (10Krinkle) Tentatively moving out... [18:38:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Reading-Infrastructure-Team-Backlog, 10Wikimedia-production-error (Shared Build Failure): Jenkins jobs for MediaWiki failing with 'npm: shasum check failed' - https://phabricator.wikimedia.org/T203506 (10Krinkle) [18:40:41] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Update blubber parser to v4 - https://phabricator.wikimedia.org/T218142 (10thcipriani) [18:42:41] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10AbuseFilter, 10CX-deployments, and 2 others: mediawiki/vendor REL1_* no more ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560 (10Krinkle) >>! In T189560#4744918, @gerritbot w... [18:54:54] Can someone who has permissions help with remaining stuff on https://phabricator.wikimedia.org/T216675 please? [18:56:24] o/ Pchelolo and I want to merge a config change [18:56:29] https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/495906/ [18:56:36] not sure how that fits in with today's train [18:56:55] it looks there is no activity happening in the next hour? [18:56:59] if so, can we deploy? [18:57:10] like, when the train goes in Euro window, does that mean that american window is like an extra SWAT window? [19:02:07] twentyafterfour: ^ do you know? [19:08:22] ottomata: Pchelolo we have a break before the train on Tuesdays to ensure that the train happens on time/has time to backport any patches after branch cut, due to the deployment schedule being copy-and-paste that stays even when the train deploy moves to EU time, which is to say: it would be fine to deploy something now [19:08:56] ok great, we are coordinating with stephen in -ops then; he's got a patch too [19:27:21] 10Continuous-Integration-Infrastructure, 10Shinken: Host DOWN alert for integration-publishing02! - https://phabricator.wikimedia.org/T218146 (10hashar) [20:04:08] 10Beta-Cluster-Infrastructure, 10Discovery-Search, 10Elasticsearch, 10Wikimedia-Logstash: ApiFeatureUsage data is not being populated in the Beta Cluster - https://phabricator.wikimedia.org/T183156 (10Anomie) Did you figure out logging in to horizon? [20:28:30] (03PS1) 10Gehel: java8: also update docker image version for job templates [integration/config] - 10https://gerrit.wikimedia.org/r/495996 [20:29:01] thcipriani: if you're still around, looks like I forgot 2 more version updates ^ [20:29:59] (03PS1) 10Daimona Eaytoy: Add Flow dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/495997 [20:48:13] (03CR) 10Thcipriani: [C: 03+2] java8: also update docker image version for job templates [integration/config] - 10https://gerrit.wikimedia.org/r/495996 (owner: 10Gehel) [20:50:41] (03Merged) 10jenkins-bot: java8: also update docker image version for job templates [integration/config] - 10https://gerrit.wikimedia.org/r/495996 (owner: 10Gehel) [20:51:28] gehel: thanks for that patch, all deployed now ^ [20:58:59] thcipriani: I hope it's the last one! Thanks! [21:13:38] Hello Release Engineers, it looks like Special:Version in beta labs is broken. https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version claims that the version of GrowthExperiments deployed there is from ~24h ago, but we've experimentally determined that that's not true (we recently merged a patch and that code is clearly there) [21:15:07] It also claims that MW core hasn't been updated there for over three hours (17:55 UTC), that's a clear sign something is broken. And I think it's the git info stuff, because it does look like updates are succesfully making it there [21:15:29] No errors from the cache_git_info scap step in the Jenkins logs for beta-scap-eqiad though [21:15:42] (03PS1) 10Umherirrender: [CleanChanges] Add seccheck [integration/config] - 10https://gerrit.wikimedia.org/r/496020 [21:16:22] latest core commit on deployment-deploy01 changed languages/i18n/az.json [21:16:25] f7867ea98bc4962ed2bbe86d04f346558dff06230785098a57ccfeac30ddaad4 [21:16:42] krenair@deployment-mediawiki-09:~$ sha256sum /srv/mediawiki/php-master/languages/i18n/az.json [21:16:42] 6e84337a4dbe326afeb40c587ad393eea00595cea39eded76e2150428568cd8f /srv/mediawiki/php-master/languages/i18n/az.json [21:16:56] krenair@deployment-mediawiki-07:~$ sha256sum /srv/mediawiki/php-master/languages/i18n/az.json [21:16:56] 6e84337a4dbe326afeb40c587ad393eea00595cea39eded76e2150428568cd8f /srv/mediawiki/php-master/languages/i18n/az.json [21:16:57] hm [21:17:17] scap is running, might be in-flight [21:17:36] scap beta is very slow so it can take some time, yup [21:17:44] before that latest commit, the next one is timestamped 17:55:18 [21:17:57] So the core part could be right, RoanKattouw [21:18:02] * Krenair digs into GrowthExperiments [21:18:41] For GrowthExperiements I looked at "ReadingModeNamespaces" being listed in HelpPanelHooks::getModuleData() [21:19:11] latest commit timestamped 18:58:52, touches includes/HelpPanelHooks.php [21:19:14] i.e. this change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/495076/8/includes/HelpPanelHooks.php [21:19:22] 63ac66e467ecfbbe5ec36741fa4bbae943c7b57d4ce15e3e0da029c5d06f360d [21:19:40] krenair@deployment-mediawiki-09:~$ sha256sum /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/HelpPanelHooks.php [21:19:40] 63ac66e467ecfbbe5ec36741fa4bbae943c7b57d4ce15e3e0da029c5d06f360d /srv/mediawiki/php-master/extensions/GrowthExperiments/includes/HelpPanelHooks.php [21:19:51] -07 concurs [21:19:57] okay so that one is definitely live [21:20:29] Yes. And the CommitDate on that commit is 2019-03-12 18:58:52 , but Special:Version says (0af5658) 21:16, 11 March 2019 [21:20:35] yep [21:20:39] something is broken here [21:21:23] (though the core difference you also found may be a red herring) [21:21:41] worthy of a task imo [21:22:40] 10Release-Engineering-Team (Backlog), 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10Tgr) When done, the [[https://www.mediawiki.org/wiki/Review_queue#Compatibility_with_other_deployed_extensions|e... [21:32:09] 10Release-Engineering-Team (Backlog), 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10Jrbranaa) > When done, the extension review instructions should probably be updated. Good catch. Will create t... [21:36:16] 10Release-Engineering-Team (Backlog), 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Figure a way to keep usermerge log entries - https://phabricator.wikimedia.org/T218160 (10MarcoAurelio) [21:37:11] 10Release-Engineering-Team (Backlog), 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10MarcoAurelio) >>! In T216089#4954141, @MarcoAurelio wrote: > We should probably figure a way to keep `Special:Lo... [22:24:42] 10Beta-Cluster-Infrastructure, 10Elasticsearch, 10Wikimedia-Logstash, 10Discovery-Search (Current work): ApiFeatureUsage data is not being populated in the Beta Cluster - https://phabricator.wikimedia.org/T183156 (10EBernhardson) a:03EBernhardson [22:25:54] greg-g: noticed a few minor errors with prod job queue due to ukwikimedia wiki being deleted. Looks like https://wikitech.wikimedia.org/wiki/Delete_a_wiki wasn't followed (specifically, I suspect globalusage wasn't cleared on commons, as deleteWiki.php would do, and other steps may be forgotten as well). [22:26:22] I can't seem to find who led the deletion of that wiki. Seems to have scattered tasks from 2016 through to 2018 [22:26:32] https://phabricator.wikimedia.org/T169488 / https://phabricator.wikimedia.org/T168436 [22:26:34] Krinkle: huh, crap. I don't know off the top of my head when that wiki was deleted.... [22:26:54] but.. someone should go through that list and make sure things are correct to avoid random issues. [22:26:59] yeah.... [22:27:14] Now I think the impact of the error I spotted was just that some commons query may be broken for a few rare images that used to be used on that wiki. [22:27:33] e.g. Special:GlobalUsage/ [22:27:43] But not sure what else would/could happen. [22:27:51] heh: https://tools.wmflabs.org/sal/production?p=0&q=ukwikimedia&d= [22:28:06] 2009: added blob_tracking table to ukwikimedia [22:28:07] * greg-g nods [22:28:07] amazing [22:28:22] Looking straight down into the centre of the Earth. [22:29:33] that's um [22:29:39] ExternalStore stuff? [22:29:40] right? [22:35:33] Project beta-scap-eqiad build #241179: 04FAILURE in 8 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241179/ [22:46:44] Yippee, build fixed! [22:46:45] Project beta-scap-eqiad build #241180: 09FIXED in 9 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241180/ [22:48:13] Project beta-scap-eqiad build #241181: 04FAILURE in 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241181/ [22:48:19] 10Deployments, 10Release-Engineering-Team (Backlog): Review removal of ukwikimedia wiki - https://phabricator.wikimedia.org/T218170 (10greg) p:05Triage→03Low [22:48:25] Krinkle: ^ [22:48:44] Krenair: way back yeah, used to be. [22:49:01] it's been removed long ago in code, and much less long ago in prod dbs as well. [22:50:51] right so this is from the time when what is now on the separate ExternalStore DB hosts in prod were just mixed in with what are now essentially the metadata s* shards [22:51:00] ... come the thought of it isn't this the beta setup [22:51:02] heh [23:01:44] Krenair: hm.. I don't know. do we not use pc and es hosts in beta? [23:01:56] (parser cache, external store) [23:02:18] at least virtual ones (I mean, they're all virtual, but like, different db on the same instance) [23:02:36] I don't think we do [23:04:04] Yippee, build fixed! [23:04:05] Project beta-scap-eqiad build #241182: 09FIXED in 9 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241182/ [23:04:32] Krinkle: [23:04:37] alex@alex-laptop:/$ ssh root@deployment-db04 -t "mysql enwiki -e 'show tables;'" | grep blobs [23:04:37] | blobs1 | [23:04:37] | blobs_flow1 | [23:04:37] Connection to deployment-db04.deployment-prep.eqiad.wmflabs closed. [23:05:12] pretty sure that's ES stuff [23:05:24] Hm.. right [23:05:28] But it's blobs, not text. [23:05:36] so that's almost what I proposed, but not exactly. [23:05:58] sure it's the modern schema with the old prod hosting style right [23:05:59] it means we do use ES in beta, but with its tables directly in the same database. [23:07:09] Basically, yeah, but not because it's old. It was set up this way to mirror production without an extra db instance. [23:07:21] Whether it's on the enwiki db or a separate one doesn't really matter that much. [23:07:52] afaik beta was created after itd been years since prod used it. [23:07:56] it=non-es [23:32:55] 10Beta-Cluster-Infrastructure, 10Puppet: puppetmaster config in deployment-prep may be inadvertently breaking store,logstash reports? - https://phabricator.wikimedia.org/T218175 (10Krenair) [23:35:57] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/VideoCutTool] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/496075 [23:35:59] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [labs/tools/VideoCutTool] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/496075 (owner: 10QChris) [23:36:20] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/VideoCutTool] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/496076 [23:36:22] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [labs/tools/VideoCutTool] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/496076 (owner: 10QChris)