[00:30:36] 10Gerrit, 10Release-Engineering-Team: Upgrade gerrit to 2.15.4 - https://phabricator.wikimedia.org/T205784 (10Paladox) [04:05:23] 10Continuous-Integration-Infrastructure (shipyard): Don't enable XDebug by default in composer-package-php70-docker jobs - https://phabricator.wikimedia.org/T188363 (10Krinkle) [04:05:33] 10Continuous-Integration-Infrastructure (shipyard): XDebug should be disabled in composer-package-php70-docker jobs - https://phabricator.wikimedia.org/T188363 (10Krinkle) [04:06:10] 10Continuous-Integration-Infrastructure: Disable xdebug's html formatting of PHP errors for Apache on Jenkins slaves - https://phabricator.wikimedia.org/T97040 (10Krinkle) 05Open>03declined [04:27:00] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests: Coverage report wrongly marks some lines of a multi-line function call as uncovered - https://phabricator.wikimedia.org/T205812 (10Krinkle) [04:27:27] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests: Coverage report wrongly marks some lines of a multi-line function call as uncovered - https://phabricator.wikimedia.org/T205812 (10Krinkle) [05:24:21] !log deployment-mcs01 deployed [mobileapps/deploy@07cbfb4]: Update mobileapps to a1fa41b [05:24:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:10:03] !log deployment-maps04 updated kartotherian and tilerator to latest [07:10:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:16:44] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash, 10WorkType-NewFunctionality: Create a logstash input filter to preprocess mysqld syslog messages - https://phabricator.wikimedia.org/T140751 (10Marostegui) [08:17:27] 10Phabricator: AphrontCountQueryException when trying to visit my Profile in Phabricator - https://phabricator.wikimedia.org/T205817 (10Aklapper) [09:21:22] Project beta-update-databases-eqiad build #28720: 04FAILURE in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28720/ [09:30:20] 10Gerrit, 10MediaWiki-extensions-General: [Clean Up] Check archived extensions on Gerrit and update their corresponding extension docs page (or visa versa) - https://phabricator.wikimedia.org/T205835 (10Mainframe98) [09:45:03] 10Gerrit, 10MediaWiki-extensions-General, 10Need-volunteer: [Clean Up] Check archived extensions on Gerrit and update their corresponding extension docs page (or visa versa) - https://phabricator.wikimedia.org/T205835 (10D3r1ck01) [10:05:40] 10Deployments, 10MediaWiki-Internationalization, 10Patch-For-Review, 10Performance-Team (Radar): Experiment with plain .php files for l10n cache instead of CDB - https://phabricator.wikimedia.org/T99740 (10Seb35) I’ve created some statistics with this proposed method. Summary: * generation time: 5x quicker... [10:21:56] Yippee, build fixed! [10:21:57] Project beta-update-databases-eqiad build #28721: 09FIXED in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28721/ [10:33:01] Hi! [10:33:13] Matt here from Search [10:33:21] I'm with Guillaume [10:33:39] I need some help/clarification with scap [10:34:24] is there a way I can deploy to just a single dsh target? instead of using the --limit-hosts option, I just specify the dsh target and it deploys there.. [11:30:12] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-MModell, 10User-Ryasmeen: 404 on workboard for an existing project (due to custom filter applied which did not exist in database) - https://phabricator.wikimedia.org/T199207 (10Aklapper) https://discourse.phabricator-community.org/t/workboard-404-a... [13:07:56] 10Scap: [scap] Log directly to logstash via syslog input - https://phabricator.wikimedia.org/T86969 (10fgiunchedi) [13:25:14] 10Release-Engineering-Team (Watching / External), 10Core Platform Team Kanban (Kanban), 10Epic, 10User-notice: Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733 (10CCicalese_WMF) [13:32:35] 10Continuous-Integration-Infrastructure, 10Wikimedia-General-or-Unknown, 10Core Platform Team Kanban (Kanban), 10Patch-For-Review, 10Performance-Team (Radar): Deprecate/obsolete $wgWikimediaJenkinsCI - https://phabricator.wikimedia.org/T200650 (10CCicalese_WMF) a:03Legoktm [13:45:48] 10Phabricator, 10Pywikibot: pywikibot tag isn't available in Tag list - https://phabricator.wikimedia.org/T205860 (10Dvorapa) [13:46:06] 10Phabricator, 10Pywikibot: pywikibot tag isn't available in Tag list - https://phabricator.wikimedia.org/T205860 (10Dvorapa) [13:47:57] 10Phabricator, 10Pywikibot: pywikibot tag isn't available in Tag list - https://phabricator.wikimedia.org/T205860 (10Dvorapa) I can not reproduce this, every time I write "Pyw", I get "Pywikibot" suggested as the first option [13:48:11] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash: Logstash in beta doesn't have any logs - https://phabricator.wikimedia.org/T205863 (10Ladsgroup) [14:52:01] 10Phabricator, 10Pywikibot: pywikibot tag isn't available in Tag list - https://phabricator.wikimedia.org/T205860 (10Aklapper) Cannot reproduce but the problem sometimes exists indeed, see e.g. T175755 [15:17:18] 10Phabricator, 10Pywikibot: pywikibot tag isn't available in Tag list - https://phabricator.wikimedia.org/T205860 (10Dvorapa) (Could both tasks be duplicates?) [15:24:42] 10Phabricator (Upstream), 10Pywikibot, 10Upstream: pywikibot tag isn't available in Tag list - https://phabricator.wikimedia.org/T205860 (10Dvorapa) [15:24:44] 10Phabricator (Upstream), 10Pywikibot, 10Upstream: pywikibot tag isn't available in Tag list - https://phabricator.wikimedia.org/T205860 (10Xqt) >>! In T205860#4630148, @Dvorapa wrote: > (Could both tasks be duplicates?) Don't know. The other task works for me. [15:25:27] 10Phabricator (Upstream), 10Pywikibot, 10Upstream: pywikibot tag isn't available in Tag list - https://phabricator.wikimedia.org/T205860 (10Dvorapa) For me the other task not :D [15:28:28] 10Release-Engineering-Team (Next), 10Scap, 10WMF-JobQueue, 10Core Platform Team Kanban (Watching / External), and 2 others: Add jobrunner servers to Scap canary process - https://phabricator.wikimedia.org/T172480 (10CCicalese_WMF) [16:44:51] 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Core Platform Team Kanban (Watching / External), 10Patch-For-Review: What happens if the Wikibase project specifies a version of a library outside of the range included in mediawiki-vendor? - https://phabricator.wikimedia.org/T179663 (10CCicale... [16:52:36] !log removing old workspace src directories left by non-quibble docker jobs on integration-slave-docker-1041 [16:52:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:12:41] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Disk-space related issues still occurring for Docker based CI jobs - https://phabricator.wikimedia.org/T205902 (10dduvall) [17:13:24] !log bringing integration-slave-docker-1041 back online following source directory clean up (T205902) [17:13:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:13:29] T205902: Disk-space related issues still occurring for Docker based CI jobs - https://phabricator.wikimedia.org/T205902 [17:15:40] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Disk-space related issues still occurring for Docker based CI jobs - https://phabricator.wikimedia.org/T205902 (10dduvall) p:05Triage>03High a:03dduvall [17:22:53] Project beta-update-databases-eqiad build #28728: 04FAILURE in 2 min 53 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28728/ [17:22:53] Project beta-scap-eqiad build #223605: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223605/ [17:44:58] Yippee, build fixed! [17:44:58] Project beta-scap-eqiad build #223606: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223606/ [18:22:14] Yippee, build fixed! [18:22:15] Project beta-update-databases-eqiad build #28729: 09FIXED in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28729/ [18:53:05] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Services: Track and install additional npm packages for all service container images - https://phabricator.wikimedia.org/T205911 (10thcipriani) [18:54:06] Hi there. I think RelEng manages the Sauce Labs plan. Would it be possible to add my account, "niedzielski"? [18:54:52] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Services: Track and install additional npm packages for all service container images - https://phabricator.wikimedia.org/T205911 (10Pchelolo) [18:58:29] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [18:59:10] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Services: Track and install additional npm packages for all service container images - https://phabricator.wikimedia.org/T205911 (10Pchelolo) There is a base set of npm packages that are used by all services. Currently, server.js installs heap... [18:59:53] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Services (watching): Track and install additional npm packages for all service container images - https://phabricator.wikimedia.org/T205911 (10Pchelolo) [19:11:24] Project beta-scap-eqiad build #223612: 04FAILURE in 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223612/ [19:11:24] Project mediawiki-core-doxygen-docker build #1608: 04FAILURE in 7 min 19 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/1608/ [19:26:14] Project beta-scap-eqiad build #223613: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223613/ [19:39:25] Project beta-scap-eqiad build #223614: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223614/ [19:40:12] 10Continuous-Integration-Infrastructure, 10Core Platform Team Kanban (Kanban), 10PHP 7.2 support: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10CCicalese_WMF) [19:44:42] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: TEC3:O1.1:Q1 Goal - Move verify stage from Minikube to CI k8s namespace in production context - https://phabricator.wikimedia.org/T199742 (10thcipriani) 05Open>03Resolved [19:50:46] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Services: TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline - https://phabricator.wikimedia.org/T205919 (10thcipriani) [19:51:39] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Services: TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline - https://phabricator.wikimedia.org/T205919 (10thcipriani) p:05Triage>03Normal [19:52:32] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline, 10Services (watching): Move Graphoid to Kubernetes via the deployment pipeline - https://phabricator.wikimedia.org/T203091 (10thcipriani) [19:52:41] Project beta-scap-eqiad build #223615: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223615/ [19:53:27] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Services: TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline - https://phabricator.wikimedia.org/T205919 (10thcipriani) [19:53:30] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline, 10Epic, 10Services (watching): Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10thcipriani) [19:54:51] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Services: TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline - https://phabricator.wikimedia.org/T205919 (10thcipriani) [19:57:44] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Blubberoid – create swagger spec - https://phabricator.wikimedia.org/T205920 (10thcipriani) [19:58:09] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Blubberoid – create swagger spec - https://phabricator.wikimedia.org/T205920 (10thcipriani) p:05Triage>03Normal [19:58:39] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Blubberoid – create swagger spec - https://phabricator.wikimedia.org/T205920 (10thcipriani) [19:58:42] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Services: TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline - https://phabricator.wikimedia.org/T205919 (10thcipriani) [20:06:01] Project beta-scap-eqiad build #223616: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223616/ [20:09:53] !log deployment-deploy01:sudo rm -rf /tmp/scap_l10n_* to remove stale l10n json and free up space [20:09:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:12:52] Yippee, build fixed! [20:12:53] Project mediawiki-core-doxygen-docker build #1609: 09FIXED in 8 min 46 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/1609/ [20:16:44] 10Gerrit, 10Release-Engineering-Team: Upgrade gerrit to 2.15.4 - https://phabricator.wikimedia.org/T205784 (10Paladox) [20:20:53] Yippee, build fixed! [20:20:54] Project beta-scap-eqiad build #223617: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223617/ [20:27:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: TEC3:O1:O1.2 Goal – Formalize the collection of CI infrastructure and tooling metrics - https://phabricator.wikimedia.org/T205923 (10dduvall) [20:27:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: TEC3:O1:O1.2 Goal – Formalize the collection of CI infrastructure and tooling metrics - https://phabricator.wikimedia.org/T205923 (10dduvall) p:05Triage>03Normal [20:52:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Collect and expose Jenkins build metrics for visualization, reporting, and analysis - https://phabricator.wikimedia.org/T205927 (10dduvall) p:05Triage>03Normal [20:54:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Collect and expose Jenkins build metrics for visualization, reporting, and analysis - https://phabricator.wikimedia.org/T205927 (10dduvall) [20:54:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Goal, and 2 others: Add Prometheus exporter to Jenkins instances - https://phabricator.wikimedia.org/T182759 (10dduvall) [20:55:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Goal, and 2 others: Add Prometheus exporter to Jenkins instances - https://phabricator.wikimedia.org/T182759 (10dduvall) a:03dduvall [21:07:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Goal, and 2 others: Add Prometheus exporter to Jenkins instances - https://phabricator.wikimedia.org/T182759 (10dduvall) As of this morning both Jenkins master have the Prometheus plugin installed and enabled. The... [21:12:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Set up a Prometheus server under the integration project - https://phabricator.wikimedia.org/T205930 (10dduvall) p:05Triage>03Normal [21:14:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Set up a Prometheus server under the integration project - https://phabricator.wikimedia.org/T205930 (10dduvall) See {T182759} for the correct Prometheus target URLs of the Jenkins masters. [21:30:57] (03PS1) 10Dduvall: Wipe entire workspace after builds of Docker based jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) [21:34:11] hi relengers! [21:34:27] FR-tech is still getting failures on all the patches in the civicrm repo [21:34:51] e.g. https://gerrit.wikimedia.org/r/463846 [21:35:08] no way should those changes yield a ton of DB errors [21:35:37] Some from last week look like out-of-memory errors [21:35:56] did the CI infrastructure change recently in a way that would be limiting memory? [21:36:51] And can we get the limit raised for that repo? [21:38:35] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-MModell, 10User-Ryasmeen: 404 on workboard for an existing project (due to custom filter applied which did not exist in database) - https://phabricator.wikimedia.org/T199207 (10mmodell) There is an upstream task for this: https://secure.phabricator... [21:38:48] 10Phabricator (Upstream), 10Release-Engineering-Team (Kanban), 10Upstream, 10User-MModell, 10User-Ryasmeen: 404 on workboard for an existing project (due to custom filter applied which did not exist in database) - https://phabricator.wikimedia.org/T199207 (10mmodell) [21:45:34] (03PS1) 10Dduvall: Revert "Use a volume under workspace for /tmp in docker containers" [integration/config] - 10https://gerrit.wikimedia.org/r/463852 [21:46:19] (03CR) 10jerkins-bot: [V: 04-1] Revert "Use a volume under workspace for /tmp in docker containers" [integration/config] - 10https://gerrit.wikimedia.org/r/463852 (owner: 10Dduvall) [21:48:05] thcipriani, zeljkof, hashar, any idea what might be wrong with the tests in the wikimedia/fundraising/crm repo? [21:48:16] They're all failing lately with DB errors [21:48:26] e.g. https://gerrit.wikimedia.org/r/463846 [21:48:45] last week I saw a few failing to allocate memory [21:49:03] let me see if I can track one of those down [21:49:42] ejegg: it's midnight here, please CC me on a phab task so I don't forget tomorrow :) [21:49:57] oops, sorry! [21:50:18] (03PS2) 10Dduvall: Revert "Use a volume under workspace for /tmp in docker containers" [integration/config] - 10https://gerrit.wikimedia.org/r/463852 [22:30:25] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:17] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [22:38:51] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Services (watching): TEC3:O3:O3.1:Q2 Goal - Move Blubberoid, ZoteroV2, and Graphoid through the production CD Pipeline - https://phabricator.wikimedia.org/T205919 (10Pchelolo) [22:48:18] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/463865 [22:48:21] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/463865 (owner: 10QChris) [22:54:21] (03PS1) 10QChris: Import done. Revoke import grants [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/463867 [22:54:24] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [software/thumbor-plugins] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/463867 (owner: 10QChris) [23:26:26] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:30:42] 10Deployments, 10Release-Engineering-Team, 10Core Platform Team, 10MediaWiki-extensions-LocalisationUpdate, and 3 others: Localization Cache Redo - https://phabricator.wikimedia.org/T78802 (10CCicalese_WMF) [23:30:58] 10Deployments, 10Release-Engineering-Team, 10Core Platform Team, 10TechCom-RFC, and 2 others: RFC: Reevaluate LocalisationUpdate extension for WMF - https://phabricator.wikimedia.org/T158360 (10CCicalese_WMF) [23:31:14] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [23:32:40] 10Continuous-Integration-Infrastructure, 10Core Platform Team, 10Core Platform Team Kanban (Later), 10Test-Coverage: Migrate https://tools.wmflabs.org/coverage/mediawiki/ to CI infrastructure - https://phabricator.wikimedia.org/T182751 (10CCicalese_WMF) [23:36:41] 10Release-Engineering-Team (Watching / External), 10Core Platform Team, 10Core Platform Team Kanban (Later): Audit all existing code to ensure that any extension currently or previously adding blobs to ExternalStore has been registering a reference in the text table ... - https://phabricator.wikimedia.org/T106388 [23:37:23] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:37:31] 10MediaWiki-Codesniffer, 10Technical-Debt: Warn when a method/class has a @deprecated annotation but no call to wfDeprecated - https://phabricator.wikimedia.org/T205938 (10Jdforrester-WMF) [23:53:51] 10Differential, 10Differential-Beta, 10OOjs-Router: Add 'npm test' Jenkins job to Differential commits for oojs-router - https://phabricator.wikimedia.org/T163865 (10Legoktm) 05Open>03declined Repository was moved to Gerrit, and has CI set up the normal way.