[00:04:42] 10Gerrit, 10Labs, 10wikitech.wikimedia.org: Request to rename LegoFan4000 to MacFan4000 on WikiTech - https://phabricator.wikimedia.org/T165624#3353570 (10bd808) I changed the uid as well. ``` $ ldapmodify -v -D 'uid=novaadmin,ou=people,dc=wikimedia,dc=org' -W -f MacFan4000-part2.ldif ldap_initialize( Project beta-update-databases-eqiad build #17839: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17839/ [00:15:45] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Labs, 10wikitech.wikimedia.org: Request to rename LegoFan4000 to MacFan4000 on WikiTech - https://phabricator.wikimedia.org/T165624#3353597 (10demon) 05Open>03Resolved a:03demon Updated on Gerrit as well. [00:22:17] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:41:25] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#3353627 (10Reedy) [01:33:14] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Labs, 10Patch-For-Review: Fix ci puppet role to support stretch - https://phabricator.wikimedia.org/T166611#3353704 (10Dzahn) added stretch support for jenkins: https://gerrit.wikimedia.org/r/3... [01:34:33] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Labs, 10Patch-For-Review: Fix ci puppet role to support stretch - https://phabricator.wikimedia.org/T166611#3353705 (10Dzahn) @paladox is the original "E: Package 'libmysqlclient-dev' has no ins... [02:05:01] Project beta-update-databases-eqiad build #17841: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17841/ [02:19:10] Dejavu? [02:23:09] !log cherry-picked https://gerrit.wikimedia.org/r/#/c/354932/ onto beta puppetmaster [02:23:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:54:32] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:58:56] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Beta-Cluster-reproducible, 10Wikimedia-log-errors: dewiktionary on beta gives Undefined variable: wmgWikibaseSiteGroup - https://phabricator.wikimedia.org/T168035#3353765 (10Reedy) [03:01:27] Looks like potentially wikidata causing the aborts [03:01:35] wikidatawiki: ...have term_full_entity_id field in wb_terms table. [03:01:35] wikidatawiki: Processed up to page 1026 (Q1014) [03:01:36] * Reedy waits [03:02:39] https://wikidata.beta.wmflabs.org/wiki/Special:RecentChanges [03:02:42] Lots of logged out edits [03:05:01] Project beta-update-databases-eqiad build #17842: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17842/ [03:05:44] Project beta-scap-eqiad build #159939: 04FAILURE in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159939/ [03:09:48] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: RebuildEntityPerPage causing beta-update-databases-eqiad to abort - https://phabricator.wikimedia.org/T168036#3353770 (10Reedy) [03:10:03] !log running `mwscript extensions/Wikibase/repo/maintenance/rebuildEntityPerPage.php --wiki=wikidatawiki` in screen as root on deployment-tin for T168036 [03:10:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:10:08] T168036: RebuildEntityPerPage causing beta-update-databases-eqiad to abort - https://phabricator.wikimedia.org/T168036 [03:11:02] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: RebuildEntityPerPage causing beta-update-databases-eqiad to abort - https://phabricator.wikimedia.org/T168036#3353784 (10Reedy) That didn't take long to run.. And then logged itself as completed... [03:13:50] !log running `mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki` in screen as root on deployment-tin for T168036 [03:13:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:13:59] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: RebuildEntityPerPage causing beta-update-databases-eqiad to abort - https://phabricator.wikimedia.org/T168036#3353786 (10Reedy) Ah, seems it's actually rebuildTermSqlIndex.php And the script doesn't have any sort of resumi... [03:15:37] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort - https://phabricator.wikimedia.org/T168036#3353787 (10Reedy) [03:16:05] Yippee, build fixed! [03:16:06] Project beta-scap-eqiad build #159940: 09FIXED in 2 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159940/ [03:17:09] wut [03:34:30] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [04:05:01] Project beta-update-databases-eqiad build #17843: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17843/ [04:06:07] lol [04:18:18] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [04:53:18] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:05:01] Project beta-update-databases-eqiad build #17844: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17844/ [05:51:17] Project beta-scap-eqiad build #159952: 04FAILURE in 3 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159952/ [05:57:08] Yippee, build fixed! [05:57:08] Project beta-scap-eqiad build #159953: 09FIXED in 3 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159953/ [06:05:01] Project beta-update-databases-eqiad build #17845: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17845/ [06:05:51] Project beta-scap-eqiad build #159954: 04FAILURE in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159954/ [06:16:16] Yippee, build fixed! [06:16:16] Project beta-scap-eqiad build #159955: 09FIXED in 2 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159955/ [06:43:52] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #393: 04FAILURE in 2 hr 3 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/393/ [06:44:36] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Upstream: "git review -d XXX" doesn't work for http gerrit - https://phabricator.wikimedia.org/T100987#3353879 (10hashar) @kaldari sure thing! > fetch patches manually (eg: git fetch origin refs/changes/DE/ABCDE && git checkout FETCH_HEAD `ref/changes/DE/ABC... [06:49:53] !log script upto `Processed up to page 336425 (Q235372)`... hopefully it's finished by morning [06:49:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:55:31] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Upstream: "git review -d XXX" doesn't work for http gerrit - https://phabricator.wikimedia.org/T100987#3353888 (10hashar) git-review 1.25 is the last tagged version. It is two years old and lack a few nice feature introduced meanwhile. My recommandation is to... [07:05:01] Project beta-update-databases-eqiad build #17846: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17846/ [07:27:32] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:50:34] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Labs, 10Patch-For-Review: Fix ci puppet role to support stretch - https://phabricator.wikimedia.org/T166611#3353944 (10Paladox) This ticket can be for master too. But was manly for contint. [08:05:01] Project beta-update-databases-eqiad build #17847: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17847/ [08:14:19] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:44:38] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3354025 (10hashar) [08:46:56] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3354030 (10hashar) @demon would you mind cherry picking https://gerrit-review.googlesource.com/c/110410 to our 2.13 Gerrit package? That is the backport Paladox did for s... [08:54:19] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:55:48] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#3353627 (10Paladox) Ah, we will get that feature when we upgrade to gerrit 2.14.1. [08:57:08] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#3354040 (10Paladox) [08:57:10] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.1 - https://phabricator.wikimedia.org/T156120#3354039 (10Paladox) [09:03:09] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.1 - https://phabricator.wikimedia.org/T156120#3354046 (10Peachey88) [09:03:11] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#3354045 (10Peachey88) [09:03:26] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.1 - https://phabricator.wikimedia.org/T156120#2964669 (10Peachey88) [09:03:28] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#3353627 (10Peachey88) [09:05:01] Project beta-update-databases-eqiad build #17848: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17848/ [09:14:22] 10Continuous-Integration-Config, 10Operations, 10Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#3354056 (10hashar) [09:17:32] 10Continuous-Integration-Config, 10Operations, 10Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#3354057 (10hashar) [09:25:00] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Operations, 10Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#3354059 (10hashar) I have poked the `ops` mailing list about it. [09:32:13] 10Release-Engineering-Team, 10MediaWiki-JobRunner, 10Operations, 10Beta-Cluster-reproducible: jobrunner / jobchron systemd services are in error state after a stop - https://phabricator.wikimedia.org/T168044#3354078 (10hashar) [09:34:14] 10Release-Engineering-Team, 10MediaWiki-JobRunner, 10Operations, 10Beta-Cluster-reproducible, 10Patch-For-Review: jobrunner / jobchron systemd services are in error state after a stop - https://phabricator.wikimedia.org/T168044#3354093 (10hashar) The alternative is to have `redisJobChronService` and `red... [10:05:01] Project beta-update-databases-eqiad build #17849: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17849/ [10:41:40] 10Release-Engineering-Team, 10MediaWiki-JobRunner, 10Operations, 10Beta-Cluster-reproducible, 10Patch-For-Review: jobrunner / jobchron systemd services are in error state after a stop - https://phabricator.wikimedia.org/T168044#3354172 (10Marostegui) p:05Triage>03Normal I would rather go for for your... [10:47:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Labs, 10Labs-Infrastructure, and 2 others: Lower rate of Nodepool requests to OpenStack API - https://phabricator.wikimedia.org/T167803#3354185 (10hashar) p:05Triage>03Normal [10:48:01] 10Release-Engineering-Team (Backlog): Change notification email from jenkins-bot@wikimedia.org to releng internal list - https://phabricator.wikimedia.org/T151642#3354187 (10hashar) p:05Triage>03Low [10:49:07] it is friday [10:49:09] triage time! [10:49:20] 10Continuous-Integration-Infrastructure, 10Nodepool: Nodepool leaks instances and does not gabage collect them - https://phabricator.wikimedia.org/T151949#3354194 (10hashar) p:05Triage>03Normal [10:50:08] 10Continuous-Integration-Infrastructure, 10Nodepool, 10Upstream: Nodepool leaks instances and does not gabage collect them - https://phabricator.wikimedia.org/T151949#2833005 (10hashar) p:05Normal>03Low That is addressed upstream in several patches. However I am not sure we want to invest anytime in upg... [10:53:23] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests: MediaWiki qunit Jenkins jobs should emit a junit result and process it - https://phabricator.wikimedia.org/T153759#3354202 (10hashar) p:05Triage>03Normal [10:54:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review, 10Zuul: Freshly provisionned zuul fails connecting to Gerrit due to ssh key host - https://phabricator.wikimedia.org/T157912#3354203 (10hashar) There is a couple patches for this though I haven't pushed to get them rev... [10:54:24] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10Zuul: Freshly provisionned zuul fails connecting to Gerrit due to ssh key host - https://phabricator.wikimedia.org/T157912#3354205 (10hashar) [10:54:26] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team: MobileFrontend Chrome browser test job has become unstable - https://phabricator.wikimedia.org/T167994#3354206 (10zeljkofilipin) As far as I know, nothing has changed recently. @hashar could know more. I took a look at last 3 failed builds: - 453 ha... [10:54:36] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10Zuul: Freshly provisionned zuul fails connecting to Gerrit due to ssh key host - https://phabricator.wikimedia.org/T157912#3020331 (10hashar) p:05Triage>03Low [10:55:23] 10Continuous-Integration-Infrastructure, 10Zuul: zuul-merger fails when repository names overlaps - https://phabricator.wikimedia.org/T157818#3354210 (10hashar) p:05Triage>03Low [10:55:33] 10Continuous-Integration-Config, 10Composer: integration/composer need a integration-composer-check-hhvm job - https://phabricator.wikimedia.org/T158845#3354211 (10hashar) p:05Triage>03Low [10:55:49] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10media-storage, 10Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3354212 (10hashar) p:05Triage>03Normal [10:55:57] 10Release-Engineering-Team (Next), 10Quibble: Start mysql in quibble container - https://phabricator.wikimedia.org/T166145#3354213 (10hashar) p:05Triage>03Normal [10:56:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Nodepool: Change time at which Nodepool refresh the images - https://phabricator.wikimedia.org/T166889#3354214 (10hashar) p:05Triage>03Low [10:59:45] 10Release-Engineering-Team (Watching / External), 10MediaWiki-API, 10Parsoid, 10monitoring, and 2 others: API action=parsoid-batch not available on Graphite - https://phabricator.wikimedia.org/T152776#3354229 (10hashar) p:05Triage>03Low [11:00:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Next): deployment-tin has disk space issues - https://phabricator.wikimedia.org/T166492#3354235 (10hashar) p:05Triage>03Normal [11:01:03] 10Beta-Cluster-Infrastructure, 10media-storage: On beta enable swift statsd metric - https://phabricator.wikimedia.org/T161084#3354237 (10hashar) I got them disabled via parent task T160990. [11:01:54] 10Beta-Cluster-Infrastructure, 10media-storage: On beta enable swift statsd metric - https://phabricator.wikimedia.org/T161084#3354239 (10hashar) p:05Triage>03Low [11:02:28] 10Continuous-Integration-Config: split mediawiki tests in unit/integration/smoke tests to speed up CI - https://phabricator.wikimedia.org/T162350#3354240 (10hashar) p:05Triage>03Normal [11:02:45] 10Continuous-Integration-Config, 10Release-Engineering-Team: split mediawiki tests in unit/integration/smoke tests to speed up CI - https://phabricator.wikimedia.org/T162350#3159953 (10hashar) [11:03:06] 10Release-Engineering-Team (Kanban), 10MediaWiki-JobRunner, 10Patch-For-Review: jobrunner loops every 100ms against a result cached for 1 second - https://phabricator.wikimedia.org/T162697#3354242 (10hashar) p:05Triage>03Normal a:03hashar [11:03:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Upgrade git package on zuul-merger instances contint1001 / contint2001 to benefit git-daemon - https://phabricator.wikimedia.org/T161086#3354245 (10hashar) [11:03:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Upgrade git package on zuul-merger instances contint1001 / contint2001 to benefit git-daemon - https://phabricator.wikimedia.org/T161086#3121040 (10hashar) p:05Triage>03Low [11:04:56] 10Release-Engineering-Team: Enhance debian-glue job packages validation - https://phabricator.wikimedia.org/T158553#3354248 (10hashar) One has to look at @volans list of tools in above comment ( T158553#3040333 ) [11:05:01] Project beta-update-databases-eqiad build #17850: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17850/ [11:05:06] 10Continuous-Integration-Config, 10Release-Engineering-Team: Enhance debian-glue job packages validation - https://phabricator.wikimedia.org/T158553#3354250 (10hashar) [11:05:19] 10Continuous-Integration-Config, 10Release-Engineering-Team: Enhance debian-glue job packages validation - https://phabricator.wikimedia.org/T158553#3040285 (10hashar) p:05Triage>03Low [11:05:51] 10Continuous-Integration-Infrastructure (phase-out-trusty): Migrate PHP5.5 jobs from Trusty to Jessie - https://phabricator.wikimedia.org/T161882#3354253 (10hashar) p:05Triage>03Normal [11:08:00] 10Continuous-Integration-Infrastructure, 10Nodepool: Add monitoring and capacity planning for Nodepool - https://phabricator.wikimedia.org/T113806#3354264 (10hashar) p:05Triage>03Low [11:08:44] 10Deployment-Systems, 10MediaWiki-Releasing: make-wmf-branch doesn't ensure git has proper user.name and user.email - https://phabricator.wikimedia.org/T126334#3354266 (10hashar) 05Open>03declined It is no more a concern nowadays. [11:12:56] 10Continuous-Integration-Config, 10MediaWiki-Documentation, 10Documentation: Create a Jenkins check to verify hooks.yaml formatting - https://phabricator.wikimedia.org/T116965#3354271 (10hashar) 05Open>03declined Can be reopened if one day there is activity on the parent task {T115388} [11:21:19] (03PS1) 10Hashar: Unconfigure Wikilog [integration/config] - 10https://gerrit.wikimedia.org/r/359399 (https://phabricator.wikimedia.org/T154443) [11:21:37] (03CR) 10Hashar: [C: 032] Unconfigure Wikilog [integration/config] - 10https://gerrit.wikimedia.org/r/359399 (https://phabricator.wikimedia.org/T154443) (owner: 10Hashar) [11:22:38] (03Merged) 10jenkins-bot: Unconfigure Wikilog [integration/config] - 10https://gerrit.wikimedia.org/r/359399 (https://phabricator.wikimedia.org/T154443) (owner: 10Hashar) [11:23:22] jdlrobson: around for localsettings.php question? [11:25:28] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3354338 (10zeljkofilipin) >>! In T164721#3349109, @Jdlrobson wrote: > @zeljkofilipin are you including LocalS... [11:27:13] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Wikilog, and 2 others: Archive the "Wikilog" extension (has moved toGitHub) - https://phabricator.wikimedia.org/T154443#3354342 (10hashar) 05Open>03Resolved a:03hashar The extension is gone from CI/... [11:29:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Jenkins: Update jenkins to 2.46.3 - https://phabricator.wikimedia.org/T166415#3354349 (10hashar) p:05Triage>03Normal [11:30:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Jenkins: Update jenkins to 2.46.3 - https://phabricator.wikimedia.org/T166415#3295717 (10hashar) Mostly tiny bug fixes and apparently none impacting us. Still worth upgrade to closely follow upstream, but it is not urgent either. [11:41:19] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3354369 (10zeljkofilipin) I have found how Ruby CI does it in [[ https://phabricator.wikimedia.org/diffusion/... [12:05:01] Project beta-update-databases-eqiad build #17851: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17851/ [12:11:11] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T167893#3354476 (10mmodell) [12:37:02] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Wikilog, and 2 others: Archive the "Wikilog" extension (has moved toGitHub) - https://phabricator.wikimedia.org/T154443#2911536 (10Kghbln) > So in short: lets clear out Wikilog from Gerrit, CI, translatew... [12:53:57] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:04:31] 10Release-Engineering-Team (Kanban), 10Phabricator: "Details" box in tasks should be hidden if empty - https://phabricator.wikimedia.org/T168058#3354694 (10mmodell) 05Open>03Resolved a:03mmodell This isn't upstream, it's a side-effect of an extension I've been working on. [13:04:42] 10Release-Engineering-Team (Kanban), 10Phabricator: "Details" box in tasks should be hidden if empty - https://phabricator.wikimedia.org/T168058#3354699 (10mmodell) 05Resolved>03Open Whoops, not resolved. [13:05:01] Project beta-update-databases-eqiad build #17852: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17852/ [13:05:48] 10Continuous-Integration-Infrastructure, 10Operations, 10Patch-For-Review: CI for operations/puppet is taking too long - https://phabricator.wikimedia.org/T166888#3354713 (10hashar) [13:05:50] 10Continuous-Integration-Config: Trigger a castor save when operations/puppet change are merged and change dependencies - https://phabricator.wikimedia.org/T168063#3354701 (10hashar) [13:06:01] Project beta-scap-eqiad build #159998: 04FAILURE in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159998/ [13:15:05] 10Release-Engineering-Team (Kanban), 10Phabricator: "Details" box in tasks should be hidden if empty - https://phabricator.wikimedia.org/T168058#3354724 (10mmodell) 05Open>03Resolved [13:16:30] Yippee, build fixed! [13:16:30] Project beta-scap-eqiad build #159999: 09FIXED in 2 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/159999/ [13:18:33] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<33.33%) [13:20:15] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [13:24:02] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:25:38] PROBLEM - Puppet errors on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:28:52] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3354748 (10zeljkofilipin) @Jdlrobson I have added include_once "$IP/extensions/RelatedArticles/tests/bro... [13:33:59] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [13:36:22] puppet errors will be fixed with: https://gerrit.wikimedia.org/r/#/c/359420/ [13:38:52] PROBLEM - Puppet errors on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:39:10] PROBLEM - Puppet errors on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:39:26] ^ that should recover now :) yep [13:43:06] (03PS4) 10Zfilipin: Cleaned up Ruby jobs [integration/config] - 10https://gerrit.wikimedia.org/r/356413 (https://phabricator.wikimedia.org/T164479) [13:44:50] PROBLEM - Puppet errors on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:46:27] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #430: 04FAILURE in 2 min 26 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/430/ [13:54:12] (03PS1) 10Hashar: Castor save operations/puppet postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/359429 (https://phabricator.wikimedia.org/T168063) [13:56:14] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Wikilog, and 2 others: Archive the "Wikilog" extension (has moved toGitHub) - https://phabricator.wikimedia.org/T154443#3354865 (10Aklapper) Thank you everybody! [13:57:41] (03CR) 10Hashar: [C: 032] Castor save operations/puppet postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/359429 (https://phabricator.wikimedia.org/T168063) (owner: 10Hashar) [14:00:16] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [14:01:32] (03Merged) 10jenkins-bot: Castor save operations/puppet postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/359429 (https://phabricator.wikimedia.org/T168063) (owner: 10Hashar) [14:04:01] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:01] Project beta-update-databases-eqiad build #17853: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17853/ [14:05:37] RECOVERY - Puppet errors on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [14:09:04] 10Continuous-Integration-Infrastructure, 10Operations, 10Patch-For-Review: CI for operations/puppet is taking too long - https://phabricator.wikimedia.org/T166888#3354917 (10hashar) [14:09:07] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Trigger a castor save when operations/puppet change are merged and change dependencies - https://phabricator.wikimedia.org/T168063#3354913 (10hashar) 05Open>03Resolved a:03hashar I have manually simulated the reac... [14:11:28] (03PS1) 10Hashar: Drop castor-save from operations/puppet test job [integration/config] - 10https://gerrit.wikimedia.org/r/359439 (https://phabricator.wikimedia.org/T166888) [14:16:23] (03CR) 10Hashar: [C: 032] Drop castor-save from operations/puppet test job [integration/config] - 10https://gerrit.wikimedia.org/r/359439 (https://phabricator.wikimedia.org/T166888) (owner: 10Hashar) [14:17:19] (03PS1) 10Zfilipin: Run mediawiki-core-qunit-selenium-jessie for Math and VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/359440 (https://phabricator.wikimedia.org/T164721) [14:18:43] (03Merged) 10jenkins-bot: Drop castor-save from operations/puppet test job [integration/config] - 10https://gerrit.wikimedia.org/r/359439 (https://phabricator.wikimedia.org/T166888) (owner: 10Hashar) [14:18:51] RECOVERY - Puppet errors on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [14:19:13] RECOVERY - Puppet errors on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [14:19:51] RECOVERY - Puppet errors on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [14:20:20] 10Continuous-Integration-Infrastructure, 10Operations, 10Patch-For-Review: CI for operations/puppet is taking too long - https://phabricator.wikimedia.org/T166888#3354931 (10hashar) > Faidon wrote: >> Runs xUnit and castor-save, both of which emit failures in the logs ("Result was FAILURE") (3s). > castor-sa... [14:23:59] (03CR) 10Zfilipin: [C: 032] Run mediawiki-core-qunit-selenium-jessie for Math and VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/359440 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [14:25:26] (03CR) 10jerkins-bot: [V: 04-1] Run mediawiki-core-qunit-selenium-jessie for Math and VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/359440 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [14:29:31] (03CR) 10Zfilipin: "@hashar I have no clue why the job failed :(" [integration/config] - 10https://gerrit.wikimedia.org/r/359440 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [14:32:08] zeljkof: something something is broken [14:32:09] 00:00:02.027 ValueError: Unknown remainder ['-8-gcbc7f62-wmf4jessie1'] in '2.5.0-8-gcbc7f62-wmf4jessie1' [14:32:16] zeljkof: that is due to python-pbr which is no more the proper version [14:32:25] will look at it in a few but there must be a task about it [14:33:01] hashar i guess we should just change the version number to match python-pbr [14:33:17] hashar: https://phabricator.wikimedia.org/T162787 [14:33:18] found it [14:33:52] and the fix is https://phabricator.wikimedia.org/T153877 [14:36:46] $ apt-cache policy python-pbr [14:36:46] python-pbr: [14:36:46] Installed: 1.10.0-1 [14:36:46] Candidate: 0.8.2-1 [14:36:48] Package pin: 0.8.2-1 [14:36:48] the host is wrong [14:37:24] Start-Date: 2017-06-16 06:48:11 [14:37:24] Commandline: /usr/bin/unattended-upgrade [14:37:24] Upgrade: python-pbr:amd64 (0.8.2-1, 1.10.0-1) [14:37:24] bah [14:37:59] hashar lets do 2.5.0-8.wl4? [14:40:50] !log integration-slave-jessie-1001 apt-get upgrade to downgrade python-pbr to 0.8.2 as pinned since T153877. /usr/bin/unattended-upgrade magically upgraded it for some reason [14:40:53] zeljkof: should be good now [14:40:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:40:55] T153877: Zuul has started failing on some repo's in gerrit.wikimedia.org - https://phabricator.wikimedia.org/T153877 [14:50:59] hashar: that was quick [14:52:10] (03PS1) 10Hashar: castor: save pip HTTP cache [integration/config] - 10https://gerrit.wikimedia.org/r/359441 [14:52:43] (03CR) 10Zfilipin: [C: 032] Run mediawiki-core-qunit-selenium-jessie for Math and VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/359440 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [14:52:54] (03PS2) 10Hashar: castor: save pip HTTP cache [integration/config] - 10https://gerrit.wikimedia.org/r/359441 (https://phabricator.wikimedia.org/T168063) [14:53:24] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Trigger a castor save when operations/puppet change are merged and change dependencies - https://phabricator.wikimedia.org/T168063#3355086 (10hashar) https://gerrit.wikimedia.org/r/359441 is because I found out that cas... [14:53:41] (03CR) 10Hashar: [C: 032] castor: save pip HTTP cache [integration/config] - 10https://gerrit.wikimedia.org/r/359441 (https://phabricator.wikimedia.org/T168063) (owner: 10Hashar) [14:54:27] (03Merged) 10jenkins-bot: Run mediawiki-core-qunit-selenium-jessie for Math and VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/359440 (https://phabricator.wikimedia.org/T164721) (owner: 10Zfilipin) [14:54:32] \O/ [14:54:50] (03Merged) 10jenkins-bot: castor: save pip HTTP cache [integration/config] - 10https://gerrit.wikimedia.org/r/359441 (https://phabricator.wikimedia.org/T168063) (owner: 10Hashar) [14:56:46] !log Reloading Zuul to deploy 18a50a707eac0bcdd88f48f2321af78ee399a4eb [14:56:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:05:01] Project beta-update-databases-eqiad build #17854: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17854/ [15:29:35] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<30.00%) [16:05:01] Project beta-update-databases-eqiad build #17855: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17855/ [16:15:16] 10Release-Engineering-Team (Kanban), 10Scap: scap: Investigate scaps handling of InitialiseSettings.php - https://phabricator.wikimedia.org/T166384#3294452 (10Seb35) Note that there is a `filemtime( "InitialiseSettings.php" )` in https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/wmf-confi... [16:27:08] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3355289 (10Jdlrobson) >>! In T164721#3354338, @zeljkofilipin wrote: >>>! In T164721#3349109, @Jdlrobson wrote... [16:27:49] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team: MobileFrontend Chrome browser test job has become unstable - https://phabricator.wikimedia.org/T167994#3355291 (10hashar) `unexpected HTTP response (503) (MediawikiApi::HttpError)` that means the target wiki has thrown a 500 error. So most probably a... [17:05:01] Project beta-update-databases-eqiad build #17856: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17856/ [17:19:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Upgrade git package on zuul-merger instances contint1001 / contint2001 to benefit git-daemon - https://phabricator.wikimedia.org/T161086#3355418 (10hashar) We should upgrade git on the Jessie CI instances as well. One benefit is updating subm... [17:24:20] Is there something about this patch that makes zuul ignore it completely? [17:24:23] https://gerrit.wikimedia.org/r/359233 [17:24:29] no test on initial upload [17:24:33] no test on 'recheck' [17:24:42] no test on PS2 or PS3 [17:25:14] ejegg maybe check layout.yaml or the test? [17:25:54] ejegg strangly zuul is rejecting you [17:25:57] it works for me [17:26:00] when doing recheck [17:26:07] did you change emails? [17:26:23] oho, I did! [17:26:41] ah [17:26:56] ejegg need to update it in layout.yaml :) [17:26:57] Do you know where to add the new address? [17:27:00] yep [17:27:07] ejegg what's your new email? [17:27:13] thanks! [17:27:22] I'm putting ejegg@ejegg.com on all the commits [17:27:26] ah [17:27:33] * paladox submits patch [17:27:46] and got tired of it changing them when I rebased via ui [17:27:50] thanks! [17:29:23] Your welcome [17:29:29] ejegg what was your prevous email? [17:29:40] eeggleston@wikimedia.org [17:29:59] so it was just covered under the wikimedia.org|de regex [17:30:00] thanks [17:30:04] ah [17:31:54] heh i got hit by that 500 again. [17:32:12] ah, I can submit the patch via normal git review [17:32:21] I think I see two places to put the address [17:32:39] Nope i already doing it :) [17:33:25] (03Draft1) 10Paladox: Update ejegg whitelisted email [integration/config] - 10https://gerrit.wikimedia.org/r/359486 [17:33:39] (03PS2) 10Paladox: Update ejegg whitelisted email [integration/config] - 10https://gerrit.wikimedia.org/r/359486 [17:33:46] ejegg ^^ [17:34:11] (03CR) 10Ejegg: [C: 031] "Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/359486 (owner: 10Paladox) [17:34:20] Your welcome :) [17:39:04] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [140.0] [17:42:29] nodepool has stopped ^^ [17:48:04] RECOVERY - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] [17:49:41] for the record: nodepool had not stopped, that's not what that warning means in any way. The backlog just got big very quickly (lots of ops/puppet patches all at once) [18:05:01] Project beta-update-databases-eqiad build #17857: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17857/ [18:44:18] wtf are wikidata doing [18:45:49] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort - https://phabricator.wikimedia.org/T168036#3355656 (10Reedy) p:05Triage>03High [18:45:54] Reedy: I was goign to ask chad if that manual run ever finished... (he's taking today off) [18:45:58] It did [18:46:07] oh, then... wtf [18:46:13] But if I run update.php again from tin [18:46:22] ...have term_full_entity_id field in wb_terms table. [18:46:22] Processed up to page 1026 (Q1014) [18:46:23] Processed up to page 2026 (Q2015) [18:46:27] It's just doing it again [18:46:29] 10Continuous-Integration-Infrastructure, 10monitoring: tune gearman alarms - https://phabricator.wikimedia.org/T168085#3355658 (10hashar) [18:46:36] And the Wikidata people seem to be ignoring me atm [18:46:41] https://phabricator.wikimedia.org/T167981 [18:46:55] :( [18:47:02] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort - https://phabricator.wikimedia.org/T168036#3353770 (10Reedy) So I ran the script... And update.php for wikidatawiki on beta... ``` ...have t... [18:47:27] I've been using ^ as the very specific cause [18:47:44] * greg-g nods [18:48:02] But I've got a couple of open wikidata tasks for a few things [18:48:07] And no one seems to be triaging or responding at all [18:48:09] Which is weird [18:48:41] huh [18:49:24] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort - https://phabricator.wikimedia.org/T168036#3355686 (10Reedy) [18:49:26] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban): beta-update-databases-eqiad times out on wikidatawiki (maintenance update) - https://phabricator.wikimedia.org/T167981#3355685 (10Reedy) [18:50:44] I might just hack it out in repo/includes/Store/Sql/DatabaseSchemaUpdater.php in both repos [18:51:29] https://github.com/wikimedia/mediawiki-extensions-Wikidata/blob/master/extensions/Wikibase/repo/includes/Store/Sql/DatabaseSchemaUpdater.php [18:51:45] Yeah, that seems to be just running unconditionally, every time [18:54:09] greg-g: There's 2 or 3 simultaneous runs of update.php on wikidatawiki [18:54:13] who owns the github mirrors for wikimedia repos? I have some pull reqs there in https://github.com/wikimedia/wikidata-query-rdf/pulls that are already implemented but I can't close them. What's the policy on pull reqs in github mirrors? [18:56:01] If you want to comment on them, link to changes, we can close them [18:56:19] SMalyshev: per team policy on what to do with PR on the mirrors. And, I forget how to link to who are the admins of the github wikimedia org [18:57:36] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: rebuildTermSqlIndex.php causing beta-update-databases-eqiad to timeout and abort - https://phabricator.wikimedia.org/T168036#3355697 (10Reedy) Or is it? It's hard to tell what's running. That syntax seems to be used by mul... [18:58:22] greg-g: https://phabricator.wikimedia.org/P5589 [18:58:25] At least one is thcipriani [18:59:04] yeah, I checked on the screen you started, looked like it finished [18:59:18] I killed all the stuck ones and started a new screen [18:59:44] Reedy: should I ping somebody specific there or just comment? [18:59:47] > Processed up to page 97710 (Q75556) [19:00:02] It seems these wikidata scripts are running every fricken time [19:00:08] They're not being run as a run once [19:00:27] well that would explain the timeouts [19:00:54] I mean, sure... I can sorta understand them needing to be run in a long running process, once [19:01:07] But then... Why the script doesn't resume.. But starts again [19:01:10] Is bizzare [19:02:46] I'm gonna disable the jenkins job for db update [19:02:52] Because there is no point it running [19:03:36] Hmm. Does that cause all of scap to break? [19:03:43] If so, we just want to neuter it [19:04:07] !log disabled beta-update-databases-eqiad because it's not doing much useful atm [19:04:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:04:14] Project beta-update-databases-eqiad build #17858: 15ABORTED in 44 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17858/ [19:04:55] thcipriani: There's still two or 3 instances of update.php running against wikidatawiki [19:05:31] I'm on the server and can kill them. Any point in continuing the one I'm running? [19:05:41] Maybe, yeah [19:05:53] The other thought... Is with them being refactored, it's actually re-running various of them [19:05:58] But there's no point > 1 running [19:06:39] right, this was my plan, try to run one through to completion and then see if the job worked, didn't kill the job though :P [19:07:03] heh [19:07:36] Ok, so that's just one running now, right [19:07:36] alright, I'm going to leave that running in screen and I'll check on it in a bit [19:07:39] right [19:08:05] It's probably gonna take 5 or 6 hours per script it needs to run [19:08:23] 3.5 hours did maybe 2/3rds last night [19:09:28] wowza, alright, I'll poke it every couple hours here [19:11:46] (03CR) 10Hashar: [C: 032] Update ejegg whitelisted email [integration/config] - 10https://gerrit.wikimedia.org/r/359486 (owner: 10Paladox) [19:13:25] (03Merged) 10jenkins-bot: Update ejegg whitelisted email [integration/config] - 10https://gerrit.wikimedia.org/r/359486 (owner: 10Paladox) [19:13:46] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Labs, 10Patch-For-Review: Fix ci puppet role to support stretch - https://phabricator.wikimedia.org/T166611#3355739 (10hashar) p:05Triage>03Normal [19:14:03] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Labs, 10Patch-For-Review: Fix ci puppet role to support stretch - https://phabricator.wikimedia.org/T166611#3301885 (10hashar) [19:15:25] 10Continuous-Integration-Infrastructure, 10Electron-PDFs, 10MediaWiki-Internationalization, 10Reading-Web-Backlog (Design): Phan throws false positive when checking SpecialElectronPDF file - https://phabricator.wikimedia.org/T167995#3352413 (10Jdlrobson) [19:16:24] 10Continuous-Integration-Infrastructure, 10Electron-PDFs, 10MediaWiki-Internationalization, 10Reading-Web-Backlog (Design): Phan throws false positive when checking SpecialElectronPDF file - https://phabricator.wikimedia.org/T167995#3352413 (10Jdlrobson) It's likely to hit any extension that is subclassing... [19:16:50] 10Continuous-Integration-Infrastructure, 10Electron-PDFs, 10MediaWiki-Internationalization, 10Reading-Web-Backlog: Phan throws false positive when checking SpecialElectronPDF file - https://phabricator.wikimedia.org/T167995#3355770 (10Jdlrobson) [19:26:20] thanks hashar and paladox! [19:29:27] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog, 10Reading-Web-Kanban-Board: Create bot that automatically rebases and rebuilds patches to master - https://phabricator.wikimedia.org/T167181#3355802 (10Jdlrobson) [19:30:14] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog, 10Reading-Web-Kanban-Board: Create bot that automatically rebases and rebuilds patches to master - https://phabricator.wikimedia.org/T167181#3319993 (10Jdlrobson) a:03Jdlrobson Neat! Can get real time notifications via: ``` ssh -p 29418... [19:30:43] your welcome :) [19:54:15] (03PS1) 10Umherirrender: Fix indentation [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/359507 [20:12:56] (03CR) 10Hashar: "deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/359486 (owner: 10Paladox) [20:22:58] (03CR) 10Paladox: "Thanks :)" [integration/config] - 10https://gerrit.wikimedia.org/r/359486 (owner: 10Paladox) [20:58:56] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog, 10Reading-Web-Kanban-Board: Create bot that automatically rebases and rebuilds patches to master - https://phabricator.wikimedia.org/T167181#3356048 (10Jdlrobson) POC up and running. The script here: https://gist.github.com/jdlrobson/a2db... [21:50:18] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata: beta-update-databases-eqiad times out on wikidatawiki (maintenance update) - https://phabricator.wikimedia.org/T167981#3356147 (10greg) [22:03:38] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog, 10Reading-Web-Kanban-Board: Create bot that automatically rebases and rebuilds patches to master - https://phabricator.wikimedia.org/T167181#3356180 (10bmansurov) I haven't looked too deeply, but you may want to remove the watch job in fav... [22:04:07] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:09:21] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Phabricator, 10Technical-Debt: Replace deprecated phabricator conduit api calls in gerrit's its-phabricator plugin - https://phabricator.wikimedia.org/T159041#3356190 (10Paladox) I've updated https://gerrit-review.googlesource.com/#/c/98996/ to remove proje... [22:24:58] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Labs, 10Patch-For-Review: Fix ci puppet role to support stretch - https://phabricator.wikimedia.org/T166611#3356212 (10greg) (meta: moving to #releng-external as this is not being... [22:25:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Zuul: Migrate zuul-server behind systemd service - https://phabricator.wikimedia.org/T167845#3356215 (10greg) (meta: moving to #releng-external as this is not being worked on by... [22:25:22] Reedy: around? [22:25:29] aude: aye [22:25:57] i can look at update.php [22:27:12] We're half hoping letting it just run in a screen session will let it all finish [22:27:19] might be just that the script needs to be run manually, if a wiki has a lot items [22:27:23] ok [22:27:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Zuul: Migrate zuul-server behind systemd service - https://phabricator.wikimedia.org/T167845#3356225 (10Paladox) Your welcome :) [22:27:36] It's been going for about 3 hours on thcipriani's run [22:27:39] ok [22:27:42] Not sure when he last checked on it :P [22:27:46] I ran one of the scripts seperalte [22:27:49] *seperately [22:27:51] ideally we want to populate the new column in update.php [22:27:53] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog, 10Reading-Web-Kanban-Board: Create bot that automatically rebases and rebuilds patches to master - https://phabricator.wikimedia.org/T167181#3356226 (10Jdlrobson) I've worked it out. 5 minutes away from computer was all I needed :) [22:27:57] * thcipriani looks [22:28:11] > Processed up to page 431928 (Q298696) [22:28:15] still cranking away [22:28:20] but wb_terms is pretty big apparently on beta [22:28:30] thcipriani: thanks [22:29:07] ~40k to go [22:32:38] back in a bit... [22:33:20] We should probably delete a lot of stuff from the beta wikidatawiki [22:33:30] PROBLEM - Puppet errors on integration-slave-docker-1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:33:45] A new item and a new property created every minute is insane [22:33:48] maybe [22:34:03] probably it's selenium tests that create a lot of stuff [22:34:11] (diff | hist) . . N (Q358337)‎; 06:43 . . (+302)‎ . . ‎10.68.21.186 (talk)‎ (‎Created a new item) [22:34:11] (diff | hist) . . N (P163708)‎; 06:42 . . (+309)‎ . . ‎Selenium user (talk | contribs)‎ (‎Created a new property) [22:34:11] (diff | hist) . . N (Q358336)‎; 06:42 . . (+302)‎ . . ‎10.68.21.186 (talk)‎ (‎Created a new item) [22:34:11] (diff | hist) . . N (P163707)‎; 06:41 . . (+309)‎ . . ‎Selenium user (talk | contribs)‎ (‎Created a new property) [22:34:12] or bots [22:34:20] It's that IP or selenium [22:38:09] 10Beta-Cluster-Infrastructure, 10Wikidata: Wikidata on beta is getting too many edits - https://phabricator.wikimedia.org/T168101#3356254 (10Reedy) [22:39:42] 10Beta-Cluster-Infrastructure, 10Wikidata: 10.68.21.186 is creating items on beta Wikidata - https://phabricator.wikimedia.org/T168102#3356256 (10Reedy) [22:41:41] 10Beta-Cluster-Infrastructure, 10Wikidata: Periodically delete old items from beta wikidatawiki - https://phabricator.wikimedia.org/T168104#3356284 (10Reedy) [22:50:50] * Reedy deletes a load of articles [22:54:08] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [22:55:10] 10Beta-Cluster-Infrastructure, 10Wikidata: Prune items and properties from beta wikidatawiki - https://phabricator.wikimedia.org/T168106#3356323 (10Reedy) [22:55:21] !log deleting Q100000-Q200000 on beta wikidatawiki T168106 [22:55:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:55:26] T168106: Prune items and properties from beta wikidatawiki - https://phabricator.wikimedia.org/T168106 [23:00:06] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:00:10] lol, 150-170 deletes per minute [23:03:36] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Internal server error when accessing change - https://phabricator.wikimedia.org/T168012#3356347 (10demon) I'll have a look Monday [23:07:27] back [23:11:16] I'm just deleting all the things [23:13:30] RECOVERY - Puppet errors on integration-slave-docker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [23:15:09] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [23:17:04] :) [23:25:16] "Capability streamEvents is required to access this resource" [23:25:25] is that a gerrit per user thing? [23:25:28] i want to make use of ssh -p 29418 gerrit.wikimedia.org gerrit stream-events [23:25:48] jdlrobson: [23:25:52] you forgot to do [23:26:00] ssh -p 29418 jdlrobson@gerrit.wikimedia.org gerrit stream-events [23:26:35] paladox: not using that username [23:26:35] ssh -p 29418 barrybrowsertestbot@gerrit.wikimedia.org gerrit stream-events [23:26:37] has same problem [23:26:43] oh [23:26:43] so im guessing it's in wrong group..? [23:26:49] most likly [23:27:12] ah [23:27:36] jdlrbobson you will have to ask one of releng if you can add your bot to be able to access stretch-events [23:27:39] https://gerrit-review.googlesource.com/Documentation/cmd-stream-events.html#_access [23:27:47] Thanks, yeh. [23:27:47] "Caller must be a member of the privileged 'Administrators' group, or have been granted the 'Stream Events' global capability." [23:27:54] yeh [23:27:55] Reedy_: icanhaz for a bot? [23:27:57] All-Projects [23:28:05] jdlrobson: I dunno if I have the rights to grant it [23:28:22] https://gerrit.wikimedia.org/r/#/admin/groups/539,members [23:28:28] Reedy_ ^^ [23:28:39] my username Jdlrobson is but obviously i dont want to use that [23:28:49] or there's [23:28:49] https://gerrit.wikimedia.org/r/#/admin/groups/4,members [23:29:00] Nope, I don't have the rights to add anyone to it [23:29:02] thanks paladox that's super useful [23:29:13] your welcome :) [23:29:14] RainbowSprinkles: are you the man to ask? [23:29:26] Possibly. But he's not supposedly working today [23:29:32] Someone in ops should be able to hook you up though [23:29:39] jdlrobson is your bot in another group? [23:30:00] paladox: it's in no groups [23:30:05] ok [23:31:26] jdlrobson use your frankie bots username [23:31:35] when connecting over stream-events [23:31:44] Frankiebot jdlrobson+frankie@gmail.com [23:31:47] paladox: why? does that have access? [23:31:54] yeh [23:31:58] that's good to know :) [23:32:00] it's in this group https://gerrit.wikimedia.org/r/#/admin/groups/4,members [23:32:19] which is then included in https://gerrit.wikimedia.org/r/#/admin/groups/539,members [23:34:59] paladox: thanks! looks like i can use frankiebot for the time being :) [23:35:07] your welcome :) [23:35:56] (03PS3) 10Thcipriani: Docker for operations-puppet-tests [integration/config] - 10https://gerrit.wikimedia.org/r/357741 (https://phabricator.wikimedia.org/T166888) [23:41:25] !log also deleting a lot of Property:P* pages on beta wikidatawiki T168106 [23:41:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:41:30] T168106: Prune items and properties from beta wikidatawiki - https://phabricator.wikimedia.org/T168106 [23:51:33] heh phab repo's can now have an image [23:51:44] i wonder should we put the mw logo on mediawiki/core? :) [23:53:25] guess so :) [23:54:19] well that will make it so much easier at identifying the repo from a image as long as no other repo will use mw too :) [23:55:22] i guess if one has more "right" to it then core [23:56:32] but avoid that it turns into "these 500 hundred extensions are all missing a logo" ? [23:57:18] yep [23:57:37] Though i guess the other repo's could have words as images