[00:02:22] (03CR) 10Thcipriani: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/490950 (https://phabricator.wikimedia.org/T215175) (owner: 10Thcipriani) [00:14:07] greg-g: do you know how much memory/disk space CI build containers get? [00:17:11] not off the top of my head no [00:18:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10Krinkle) [00:18:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10Krinkle) [00:19:39] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689 (10Krinkle) [00:25:43] Project beta-scap-eqiad build #240536: 04FAILURE in 8 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240536/ [00:27:17] Project beta-scap-eqiad build #240537: 04STILL FAILING in 9.8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240537/ [00:44:27] Yippee, build fixed! [00:44:27] Project beta-scap-eqiad build #240538: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240538/ [00:46:16] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T206674 (10bd808) [00:47:49] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T206673 (10greg) [00:48:00] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster - User permissions - https://phabricator.wikimedia.org/T217874 (10DannyS712) [00:49:38] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster - User permissions - https://phabricator.wikimedia.org/T217874 (10greg) You're an administrator, you can yourself, no? [00:56:46] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster - User permissions - https://phabricator.wikimedia.org/T217874 (10Krenair) admins can't do that [00:57:39] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta cluster - User permissions - https://phabricator.wikimedia.org/T217874 (10Krenair) 05Open→03Resolved a:03Krenair [00:57:47] and now you beat me, Krenair :P [00:58:47] :P [01:10:33] 10Beta-Cluster-Infrastructure, 10MobileFrontend, 10Wikimedia-Site-requests, 10Patch-For-Review, and 2 others: Clean up beta cluster mobile configuration - https://phabricator.wikimedia.org/T213599 (10Jdlrobson) 05Open→03Resolved a:03Jdlrobson [01:10:42] 10Beta-Cluster-Infrastructure, 10MobileFrontend, 10Wikimedia-Site-requests, 10Patch-For-Review, and 2 others: Clean up beta cluster mobile configuration - https://phabricator.wikimedia.org/T213599 (10Jdlrobson) [01:17:51] (03PS1) 10MarkAHershberger: Start work on docker-driven make to produce tarballs [integration/config] - 10https://gerrit.wikimedia.org/r/495160 [01:19:22] (03CR) 10jerkins-bot: [V: 04-1] Start work on docker-driven make to produce tarballs [integration/config] - 10https://gerrit.wikimedia.org/r/495160 (owner: 10MarkAHershberger) [01:53:49] (03PS1) 10Smalyshev: Increase timeout for Blazegraph job [integration/config] - 10https://gerrit.wikimedia.org/r/495166 [01:56:01] (03PS2) 10Smalyshev: Increase timeout for Blazegraph job [integration/config] - 10https://gerrit.wikimedia.org/r/495166 (https://phabricator.wikimedia.org/T216855) [01:56:21] 10Phabricator: List of tasks show a line break between the task type label and project tags (if a task has a lot of project tags) - https://phabricator.wikimedia.org/T217371 (10Smalyshev) 05Open→03Resolved a:03Smalyshev Seems to be ok now. [01:57:02] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Wikidata-Query-Service-Sprint, and 2 others: Set up Blazegraph test suite on CI - https://phabricator.wikimedia.org/T216855 (10Smalyshev) We need to increase timeout, 30 mins is not enough. [02:00:38] Well i got multi-site working on gerrit-test3 (but fails on gerrit-test4) as kafka needs to bind to a ip that is accessible (ie no use mw vagrants ip) [02:27:43] oh [02:27:44] wow [02:27:45] https://gerrit.gerrit.wmflabs.org/r/q/status:open [02:27:48] got it working ^^ [03:25:13] Project beta-scap-eqiad build #240553: 04FAILURE in 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240553/ [03:44:12] Yippee, build fixed! [03:44:12] Project beta-scap-eqiad build #240554: 09FIXED in 9 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240554/ [03:45:31] Project beta-scap-eqiad build #240555: 04FAILURE in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240555/ [04:04:15] Yippee, build fixed! [04:04:16] Project beta-scap-eqiad build #240556: 09FIXED in 9 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240556/ [04:20:25] so i just did a commit on https://gerrit.git.wmflabs.org/r/q/status:open and it showed up on https://gerrit.gerrit.wmflabs.org/r/q/status:open [05:01:15] Project beta-scap-eqiad build #240561: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240561/ [05:12:33] Yippee, build fixed! [05:12:34] Project beta-scap-eqiad build #240562: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240562/ [05:37:02] Can someone delete https://phabricator.wikimedia.org/rGERRITDEPLOYd7852a549f6b#6495124 please? (Comment) [05:37:12] It’s spam [09:08:53] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T206674 (10hashar) 05Open→03Resolved 1.33.0-wmf.20 seems to be working fine :] [09:20:23] Project beta-update-databases-eqiad build #32268: 04FAILURE in 21 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32268/ [09:20:46] Project beta-scap-eqiad build #240584: 04FAILURE in 0.98 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240584/ [09:34:23] Yippee, build fixed! [09:34:24] Project beta-scap-eqiad build #240585: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240585/ [09:35:32] gehel: good morning :) [09:35:44] I got the java images build yesterday night. Now gotta switch the jobs to it [10:04:39] !log Updating maven based jobs to latest java8 containers | https://gerrit.wikimedia.org/r/#/c/integration/config/+/495022/ | T208938 [10:04:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:04:42] T208938: Use maven wrapper (mvnw) to build maven based project from search platform team - https://phabricator.wikimedia.org/T208938 [10:04:49] (03CR) 10Hashar: [C: 03+2] "Jobs deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/495022 (https://phabricator.wikimedia.org/T208938) (owner: 10Hashar) [10:07:03] (03Merged) 10jenkins-bot: Update Jenkins job to use latest java8 containers [integration/config] - 10https://gerrit.wikimedia.org/r/495022 (https://phabricator.wikimedia.org/T208938) (owner: 10Hashar) [10:20:18] gehel: and I have updated the jobs so they should now all support mvnw ! [10:21:22] Yippee, build fixed! [10:21:23] Project beta-update-databases-eqiad build #32269: 09FIXED in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32269/ [10:35:22] Project beta-scap-eqiad build #240591: 04FAILURE in 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240591/ [10:54:28] Yippee, build fixed! [10:54:29] Project beta-scap-eqiad build #240592: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240592/ [11:21:52] (03CR) 10Hashar: "I found bugs, some coming from previous changes :/" (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/495166 (https://phabricator.wikimedia.org/T216855) (owner: 10Smalyshev) [11:25:54] (03PS1) 10Hashar: Drop erroneous 'goals' variable for maven jobs [integration/config] - 10https://gerrit.wikimedia.org/r/495200 [11:28:52] (03PS3) 10Hashar: Timeout param for maven jobs, raise it for Blazegraph [integration/config] - 10https://gerrit.wikimedia.org/r/495166 (https://phabricator.wikimedia.org/T216855) (owner: 10Smalyshev) [11:30:10] (03CR) 10Hashar: [C: 03+2] "And it is a noop in JJB output since, well, "goals" is not expanded in the job template :)" [integration/config] - 10https://gerrit.wikimedia.org/r/495200 (owner: 10Hashar) [11:30:25] gehel: and i have found bugs iin the maven templates bah :(( [11:32:19] (03Merged) 10jenkins-bot: Drop erroneous 'goals' variable for maven jobs [integration/config] - 10https://gerrit.wikimedia.org/r/495200 (owner: 10Hashar) [11:34:05] 10Project-Admins: Create milestones of Deployments project - https://phabricator.wikimedia.org/T217843 (10MarkAHershberger) 05Open→03Invalid After some discussion, it looks like #epic tag is what I need, not milestones. Thanks for your patience. [11:36:17] !log integration: deleting old java8 docker images: sudo cumin --force 'name:docker' 'docker images|grep java8|awk "{ print $3 }"|xargs docker rmi' [11:36:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:37:23] (03CR) 10Hashar: [C: 03+2] "I have updated the job. Recheck is ongoing on https://gerrit.wikimedia.org/r/#/c/wikidata/query/blazegraph/+/495157/" [integration/config] - 10https://gerrit.wikimedia.org/r/495166 (https://phabricator.wikimedia.org/T216855) (owner: 10Smalyshev) [11:39:35] (03Merged) 10jenkins-bot: Timeout param for maven jobs, raise it for Blazegraph [integration/config] - 10https://gerrit.wikimedia.org/r/495166 (https://phabricator.wikimedia.org/T216855) (owner: 10Smalyshev) [12:14:29] hasharLunch: thanks for the cleanup ! [12:15:49] Project beta-scap-eqiad build #240600: 04FAILURE in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240600/ [12:26:15] 10Release-Engineering-Team (Kanban): Evaluate sourcehut for CI future WG - https://phabricator.wikimedia.org/T217889 (10LarsWirzenius) p:05Triage→03Normal [12:27:27] 10Release-Engineering-Team (Kanban): Check gerrithub's info in spreadsheet - https://phabricator.wikimedia.org/T217890 (10LarsWirzenius) p:05Triage→03High [12:29:11] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Discovery-Search (Current work), 10Patch-For-Review: Use maven wrapper (mvnw) to build maven based project from search platform team - https://phabricator.wikimedia.org/T208938 (10hashar) 05Open→03Resolved Should be good CI wise now. [12:34:29] Yippee, build fixed! [12:34:30] Project beta-scap-eqiad build #240601: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240601/ [12:39:24] addshore: hello, eventually I gave phan a try and got an issue to have it parse files from the root of the extension https://phabricator.wikimedia.org/T217794 :D [12:45:37] 10Release-Engineering-Team (Kanban): Implement "blubber building" CI task for GitLab-CI - https://phabricator.wikimedia.org/T217594 (10LarsWirzenius) I did a lightweight evaluation of GitLab-CI as described below. I did not evaluate it, except from the CI angle: I didn't consider it as a git hosting platform, or... [12:45:53] 10Release-Engineering-Team (Kanban): Implement "blubber building" CI task for GitLab-CI - https://phabricator.wikimedia.org/T217594 (10LarsWirzenius) 05Open→03Resolved [12:45:55] 10Release-Engineering-Team (Kanban): Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10LarsWirzenius) [12:48:27] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find out what Netsurf does for OS X builds - https://phabricator.wikimedia.org/T217674 (10LarsWirzenius) Netsurf uses real Mac hardware with OS X as CI workers. They're tighly firewalled and in their own virtual LAN, and only accessed over SSH. [12:48:36] 10Release-Engineering-Team (Kanban): Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10LarsWirzenius) [12:48:38] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find out what Netsurf does for OS X builds - https://phabricator.wikimedia.org/T217674 (10LarsWirzenius) 05Open→03Resolved [13:46:15] 10Release-Engineering-Team (Kanban): Evaluate sourcehut for CI future WG - https://phabricator.wikimedia.org/T217889 (10zeljkofilipin) Duplicate of T217852? [13:49:14] 10Release-Engineering-Team (Kanban): Evaluate sourcehut for CI future WG - https://phabricator.wikimedia.org/T217889 (10LarsWirzenius) Yes, sorry. [13:49:45] 10Release-Engineering-Team (Kanban): Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10LarsWirzenius) [13:49:47] 10Release-Engineering-Team (Kanban): Evaluate sourcehut for CI future WG - https://phabricator.wikimedia.org/T217889 (10LarsWirzenius) 05Open→03Invalid [14:20:27] Project beta-update-databases-eqiad build #32273: 04FAILURE in 26 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32273/ [14:30:04] * paladox just did https://gerrit.git.wmflabs.org/r/c/testing/test/+/2161#message-556be9c19b4b882db5afefbee232ea1a092609ff which showed up at https://gerrit.gerrit.wmflabs.org/r/c/testing/test/+/2161#message-556be9c19b4b882db5afefbee232ea1a092609ff [14:52:34] Project beta-scap-eqiad build #240614: 04FAILURE in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240614/ [15:04:32] Yippee, build fixed! [15:04:32] Project beta-scap-eqiad build #240615: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240615/ [15:06:58] 10Continuous-Integration-Infrastructure, 10Editing-team, 10MediaWiki-extensions-Scribunto: phpunit drops dead on some extension tests - https://phabricator.wikimedia.org/T217384 (10Anomie) >>! In T217384#5010061, @Smalyshev wrote: > This time the log is produced and ends with: > > ` > PHPUnitCommand] Start... [15:10:22] 10Gerrit: Create plugin that allows users to cherry pick commits even with merge conflicts (in polygerrit) - https://phabricator.wikimedia.org/T215659 (10Paladox) 05Open→03Declined Change has been merged upstream so no need for a plugin. [15:12:19] so [15:12:25] I started something earlier this week [15:12:28] but can't remember what :( [15:13:19] could someone review these ci changes https://gerrit.wikimedia.org/r/#/c/integration/config/+/493328/ and https://gerrit.wikimedia.org/r/#/c/integration/config/+/493328/ please? :) [15:13:22] gehel: SMalyshev: the BlazeGraph job eventually succeed after 1 hour and 42 minutes https://gerrit.wikimedia.org/r/#/c/wikidata/query/blazegraph/+/495157/ :))))))))))) [15:14:09] Wow! Time to buy a few more servers for the CI cluster :) [15:14:33] * gehel is wondering what is taking that much time. [15:14:52] I have not yet run the full build locally, I've always had crashing tests before that [15:15:11] it is with an empty maven cache though [15:15:30] then Stas told me during the all hands that would probably need a dedicatd Jenkins slave for that [15:15:33] with moaar ram/cpu [15:16:14] and maybe one can check with SRE to have a dedicated/baremetal machine for it [15:16:22] "just" need to have it isolated from prod [15:17:43] Project beta-scap-eqiad build #240617: 04FAILURE in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240617/ [15:21:37] Yippee, build fixed! [15:21:37] Project beta-update-databases-eqiad build #32274: 09FIXED in 1 min 36 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32274/ [15:24:12] so blazegraph passes [15:24:15] good [15:27:30] having the build pass is already a good step forward [15:28:04] we're probably not going to do tons of changes, so I'm not really sure it is worth investing in a dedicated build node, with all the complexity this probably brings [15:28:19] since it did build, RAM is not an issue [15:28:44] moar CPU might help speed up, but maven builds are usually not all that parallelizable [15:29:49] how much RAM is available on the build nodes? [15:30:48] surefire is run with forkCount=1, so more CPU is probably not going to help much (assuming most time is spent in the tests, which is usually the case) [15:31:37] Looks like we're running surefire with `-Xms512m -Xmx4g`. So if we don't have that much RAM available, we might be swapping, which would explain crappy perfs [15:34:28] Yippee, build fixed! [15:34:29] Project beta-scap-eqiad build #240618: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240618/ [15:46:09] gehel: I dont know about ram 16 or 32 GB iirc [15:46:15] but it is shared by multiple builds [15:46:23] though we can have a dedicated instance if that helps [15:46:58] gehel: we can try next week to spin off a 32 GB instance with only two executors [15:47:07] so the build would have 16/32GB available [15:47:14] and thus raise the -X settings then see what happens [15:47:23] also if it does a lot of disk, maybe we can use a tmpfs mount ;) [15:47:29] anywa [15:47:37] I am lost in what I had to do earlier this week [15:47:53] was related to a migration toward node10/npm6 but I cant quite process the review I have been given [15:48:01] so I give up for this week. Have all a good week-end! [15:51:09] PROBLEM - Puppet staleness on deployment-db04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [15:51:18] 10Gerrit, 10Release-Engineering-Team, 10Operations: Deploy multi-site plugin to cobalt and gerrit2001 - https://phabricator.wikimedia.org/T217174 (10Paladox) a:05Paladox→03None This can be deployed to prod (if there's kafka in prod). In my testing this worked really well (we only want replication from co... [15:52:57] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:54:27] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [3.0] [15:55:02] PROBLEM - Host deployment-prometheus01 is DOWN: CRITICAL - Host Unreachable (172.16.5.7) [15:58:23] Project beta-scap-eqiad build #240621: 04FAILURE in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240621/ [16:14:27] (03CR) 10Alexandros Kosiaris: "@Dduvall ping?" [blubber] - 10https://gerrit.wikimedia.org/r/492922 (https://phabricator.wikimedia.org/T205911) (owner: 10Alexandros Kosiaris) [16:14:33] Yippee, build fixed! [16:14:33] Project beta-scap-eqiad build #240622: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240622/ [16:15:56] Project beta-scap-eqiad build #240623: 04FAILURE in 1.1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240623/ [16:28:05] PROBLEM - Host integration-publishing02 is DOWN: CRITICAL - Host Unreachable (172.16.4.5) [16:28:52] 10Release-Engineering-Team (Kanban): Investigate sourcehut builds - https://phabricator.wikimedia.org/T217852 (10brennen) Working build manifest: https://git.sr.ht/~brennen/blubber/tree/sourcehut_test/.build.yml [16:35:26] Yippee, build fixed! [16:35:26] Project beta-scap-eqiad build #240624: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240624/ [16:36:15] 10Release-Engineering-Team (Kanban): Investigate sourcehut builds - https://phabricator.wikimedia.org/T217852 (10brennen) p:05Triage→03Normal [16:38:39] you can now cherry pick with merge conflicts in the ui :) (my change was merged https://gerrit-review.googlesource.com/c/gerrit/+/199730 ) [16:41:09] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Evaluate Phabricator Harbormaster - https://phabricator.wikimedia.org/T217901 (10zeljkofilipin) [16:41:29] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Evaluate Phabricator Harbormaster - https://phabricator.wikimedia.org/T217901 (10zeljkofilipin) p:05Triage→03Normal [16:44:15] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts, 10Epic: Automate LocalSettings.php creation for local-charts - https://phabricator.wikimedia.org/T217869 (10thcipriani) In case it's useful, here's what CI does: https://gerrit.wikimedia.org/r/plugins/gitiles/integration/quibble... [17:20:24] Project beta-update-databases-eqiad build #32276: 04FAILURE in 23 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32276/ [17:29:54] [18425ac161080489bc5f4318] [no req] MWException from line 475 of /srv/mediawiki/php-1.33.0-wmf.20/includes/cache/localisation/LocalisationCache.php: No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php. [17:29:54] Backtrace: [17:29:56] * Reedy is confused [17:31:07] 10Release-Engineering-Team (Kanban): Investigate sourcehut builds - https://phabricator.wikimedia.org/T217852 (10brennen) Notes: [[ https://sourcehut.org/ | sourcehut ]] is a suite of interoperable tools for code hosting, issue tracking, CI, etc., described as "the hacker's forge". The web interface is minimal... [17:32:35] 10Release-Engineering-Team (Kanban): Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10brennen) [17:32:37] 10Release-Engineering-Team (Kanban): Investigate sourcehut builds - https://phabricator.wikimedia.org/T217852 (10brennen) [17:58:13] 10Release-Engineering-Team (Kanban): Evaluate Tekton Pipeline - https://phabricator.wikimedia.org/T217912 (10dduvall) [18:16:00] o/ anyone around to help deploy a labs config change? https://gerrit.wikimedia.org/r/495256 [18:16:11] Project beta-scap-eqiad build #240633: 04FAILURE in 7.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240633/ [18:16:34] I'd do it but I still haven't sorted out my deployment keys since the great encryption disaster of 2019 [18:16:40] marktraceur: It probably won't work [18:16:52] oh, it's not as broken as it looks [18:17:07] Deploying it won't work? [18:17:23] I was looking at all the failures [18:17:28] Ah [18:17:36] * Reedy shrugs [18:18:05] If it gets less broken and someone is able to help out I'd appreciate it - we've been trying to get things working properly on beta for some time now [18:18:39] jfdi [18:18:43] see if it works [18:19:07] Reedy: Don't I need to sync the change to prod too? I don't have access at the moment [18:19:12] I'm just gonna do it [18:19:18] I believe in you [18:20:20] marktraceur: jerkins reckons it's deployed [18:20:26] https://integration.wikimedia.org/ci/view/Beta/job/beta-mediawiki-config-update-eqiad/14351/ [18:20:33] Triggered by change: 495256,6 [18:20:33] Branch: master [18:20:33] Pipeline: postmerge [18:20:33] All builds for this change set: [18:20:33] beta-mediawiki-config-update-eqiad #14351: SUCCESS [18:20:33] Reported result: SUCCESS [18:21:23] Yippee, build fixed! [18:21:24] Project beta-update-databases-eqiad build #32277: 09FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32277/ [18:29:32] Yippee, build fixed! [18:29:32] Project beta-scap-eqiad build #240634: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240634/ [18:30:12] Thanks Reedy! [18:30:21] I'll have a gander at it in a bit [18:37:55] (03CR) 10Kosta Harlan: [C: 03+1] Sonar: job template for change vs branch [integration/config] - 10https://gerrit.wikimedia.org/r/490950 (https://phabricator.wikimedia.org/T215175) (owner: 10Thcipriani) [18:42:48] Project beta-scap-eqiad build #240636: 04FAILURE in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240636/ [18:46:56] (03CR) 10Kosta Harlan: Remove requirement for properties file, run and import coverage (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/494548 (https://phabricator.wikimedia.org/T215176) (owner: 10Kosta Harlan) [18:54:23] Yippee, build fixed! [18:54:24] Project beta-scap-eqiad build #240637: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240637/ [19:07:00] Reedy: Sorry, I failed to spot a typo in that config patch. Follow-up incoming. [19:07:42] Or...oh, shoot, I still don't have the Gerrit SSH keys set up either. Stupid hard drive. [19:08:46] lol [19:08:52] need me to fix? [19:09:42] I can get it, I needed to set up the key anyway [19:18:14] Reedy: https://gerrit.wikimedia.org/r/495282 [19:18:18] #blamecormac [19:19:23] I don't actually see MediaInfoExternalEntitySearchBaseUri in any extension [19:19:32] Uhh [19:19:53] Is it very recent? [19:19:55] https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseMediaInfo/+/494996/5/extension.json [19:19:56] Yeah [19:20:13] And it's set on beta commons (to '') [19:20:18] Ah, ok, yeah [19:20:29] My repos are maybe 24H out of date [19:20:44] Thanks again :) [19:42:12] Project beta-scap-eqiad build #240642: 04FAILURE in 4.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240642/ [19:49:46] 10Phabricator: Phabricator admin rights for bawolff - https://phabricator.wikimedia.org/T217917 (10Bawolff) [19:54:27] Yippee, build fixed! [19:54:28] Project beta-scap-eqiad build #240643: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240643/ [19:55:55] 10Continuous-Integration-Infrastructure: phpunit drops dead on some extension tests - https://phabricator.wikimedia.org/T217384 (10Smalyshev) Hmm I was looking at the same log but I got different data that time. Now I see what you were seeing. Maybe it was somehow partially copied that time? No idea. Anyway, th... [20:01:39] (03PS1) 10Umherirrender: [DismissableSiteNotice] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/495298 [20:04:41] 10Release-Engineering-Team, 10Readers-Web-Backlog, 10Epic, 10MobileFrontend (MobileFrontend and MinervaNeue architecture): [EPIC] Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980 (10Jdlrobson) p:05Low→03Normal [20:08:59] (03PS1) 10Umherirrender: [ExtensionDistributor] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/495300 [20:11:51] (03PS1) 10Umherirrender: [Capiunto] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/495302 [20:13:13] (03CR) 10Legoktm: [C: 03+2] [ExtensionDistributor] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/495300 (owner: 10Umherirrender) [20:13:15] (03CR) 10Legoktm: [C: 03+2] [DismissableSiteNotice] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/495298 (owner: 10Umherirrender) [20:13:17] (03CR) 10Legoktm: [C: 03+2] [PageImages] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/493502 (owner: 10Umherirrender) [20:14:40] (03Merged) 10jenkins-bot: [ExtensionDistributor] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/495300 (owner: 10Umherirrender) [20:14:54] (03Merged) 10jenkins-bot: [DismissableSiteNotice] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/495298 (owner: 10Umherirrender) [20:14:57] (03Merged) 10jenkins-bot: [PageImages] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/493502 (owner: 10Umherirrender) [20:16:18] !log deploying https://gerrit.wikimedia.org/r/495300 https://gerrit.wikimedia.org/r/495298 https://gerrit.wikimedia.org/r/493502 [20:16:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:18:20] (03CR) 10Legoktm: [C: 03+2] [Capiunto] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/495302 (owner: 10Umherirrender) [20:19:58] (03Merged) 10jenkins-bot: [Capiunto] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/495302 (owner: 10Umherirrender) [20:20:35] !log deploying https://gerrit.wikimedia.org/r/495302 [20:20:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:28:17] so much phaaaaan [20:30:31] Project beta-scap-eqiad build #240647: 04FAILURE in 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240647/ [20:31:11] hash [20:31:21] I wonder if it's safe to rebuild quibble [20:31:55] probably not [20:44:26] Yippee, build fixed! [20:44:27] Project beta-scap-eqiad build #240648: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240648/ [21:12:45] (03PS1) 10Umherirrender: [Linter] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/495332 [21:20:45] Project beta-scap-eqiad build #240652: 04FAILURE in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240652/ [21:36:26] 10Scap, 10serviceops: Make canary wait time configurable - https://phabricator.wikimedia.org/T217924 (10jijiki) p:05Triage→03Normal [21:37:10] 10Release-Engineering-Team (Watching / External), 10Scap, 10serviceops, 10User-jijiki: Allow scap sync to deploy gradually - https://phabricator.wikimedia.org/T212147 (10jijiki) [21:37:18] Yippee, build fixed! [21:37:19] Project beta-scap-eqiad build #240653: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240653/ [21:37:46] 10Scap, 10serviceops: Make canary wait time configurable - https://phabricator.wikimedia.org/T217924 (10jijiki) [21:39:22] 10Release-Engineering-Team (Watching / External), 10Scap, 10serviceops, 10User-jijiki: Allow scap sync to deploy gradually - https://phabricator.wikimedia.org/T212147 (10jijiki) [21:40:33] (03PS1) 10Effie Mouzeli: Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) [21:42:33] (03PS2) 10Effie Mouzeli: Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) [21:43:07] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [21:45:33] (03Abandoned) 10Effie Mouzeli: WIP: Add tests on main.py, add --canary-wait-time [tools/scap] - 10https://gerrit.wikimedia.org/r/493699 (owner: 10Effie Mouzeli) [21:45:48] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [21:49:50] this is the image-diff plugin working https://gerrit.git.wmflabs.org/r/c/test/+/2502/1..2/Screenshot+2019-03-08+at+21.46.15.png [21:50:41] Project beta-scap-eqiad build #240655: 04FAILURE in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240655/ [21:51:33] 10Continuous-Integration-Infrastructure: phpunit drops dead on some extension tests - https://phabricator.wikimedia.org/T217384 (10Smalyshev) So here's another instance: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/38573/ Again, as I download mw-debug-cli.log right now, it ends with... [22:03:27] (03PS4) 10Thcipriani: Sonar: job template for change vs branch [integration/config] - 10https://gerrit.wikimedia.org/r/490950 (https://phabricator.wikimedia.org/T215175) [22:07:53] Yippee, build fixed! [22:07:53] Project beta-scap-eqiad build #240656: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240656/ [22:12:16] (03CR) 10Thcipriani: [C: 03+2] "Jobs deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/490950 (https://phabricator.wikimedia.org/T215175) (owner: 10Thcipriani) [22:13:47] 10Continuous-Integration-Infrastructure: phpunit drops dead on some extension tests - https://phabricator.wikimedia.org/T217384 (10Anomie) That one broke at around test 6893 of 19954, 34.54% of the way through the run versus 66.15% for the other one. Why such a big difference? The mw-debug-cli.log there is prob... [22:14:44] (03Merged) 10jenkins-bot: Sonar: job template for change vs branch [integration/config] - 10https://gerrit.wikimedia.org/r/490950 (https://phabricator.wikimedia.org/T215175) (owner: 10Thcipriani) [22:18:35] 10Continuous-Integration-Infrastructure: phpunit drops dead on some extension tests - https://phabricator.wikimedia.org/T217384 (10Smalyshev) As you see from the task description, the point where it breaks seems to be random - it can even break in different phpunit run. This one: https://integration.wikimedia.or... [22:18:38] !log Reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/490950/ [22:18:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:30:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: phpunit drops dead on some extension tests - https://phabricator.wikimedia.org/T217384 (10Smalyshev) [22:31:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Jenkins: phpunit drops dead on some extension tests - https://phabricator.wikimedia.org/T217384 (10Smalyshev)