[00:26:53] Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #8: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/8/ [01:35:54] Project beta-scap-eqiad build #101831: 04FAILURE in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/101831/ [01:46:00] Project beta-scap-eqiad build #101832: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/101832/ [01:55:56] Project beta-scap-eqiad build #101833: 04STILL FAILING in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/101833/ [01:57:28] Yippee, build fixed! [01:57:29] Project browsertests-Wikidata-WikidataTests-Group0-SmokeTests-linux-firefox-sauce build #42: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-Group0-SmokeTests-linux-firefox-sauce/42/ [02:05:54] Yippee, build fixed! [02:05:54] Project beta-scap-eqiad build #101834: 09FIXED in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/101834/ [02:38:19] PROBLEM - Parsoid on deployment-parsoid06 is CRITICAL: Connection refused [04:18:28] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #4: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/4/ [04:18:47] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2274189 (10mmodell) 05Open>03Resolved [04:42:02] Is there any rhyme or reason to which extensions use extension.json verses ExtensionName.php for their entrypoint? [04:44:25] (03CR) 10Legoktm: [C: 032] "hehe" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/287281 (owner: 10Addshore) [04:45:02] (03CR) 10Legoktm: [C: 032] Remove reflection that doesn't seem to be needed any more [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/287402 (owner: 10Addshore) [04:47:40] twentyafterfour: they're in the process of being migrated to extension.json [04:47:54] twentyafterfour: https://tools.wmflabs.org/extreg-wos/ [04:51:29] (03Merged) 10jenkins-bot: Add another pass test file with a namespace [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/287281 (owner: 10Addshore) [04:51:31] (03Merged) 10jenkins-bot: Change my author tag to addshore everywhere [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/287401 (owner: 10Addshore) [04:51:33] (03Merged) 10jenkins-bot: Remove reflection that doesn't seem to be needed any more [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/287402 (owner: 10Addshore) [05:01:10] (03PS1) 1020after4: generate extension-list when making a new branch [tools/release] - 10https://gerrit.wikimedia.org/r/287565 [05:02:28] (03PS2) 1020after4: Generate extension-list when making a new branch [tools/release] - 10https://gerrit.wikimedia.org/r/287565 (https://phabricator.wikimedia.org/T125678) [05:03:02] 10Deployment-Systems, 10MediaWiki-Configuration, 05MW-1.27-release-notes, 13Patch-For-Review: extension-list should live in the mediawiki branch rather than mediawiki-config - https://phabricator.wikimedia.org/T125678#2274242 (10mmodell) https://gerrit.wikimedia.org/r/#/c/287565/ [06:41:09] 10Beta-Cluster-Infrastructure, 10ContentTranslation-Deployments, 10ContentTranslation-cxserver, 05ContentTranslation-Release9, and 2 others: Shinken is warning about cxserver on deployment-cxserver03 returning 404s - https://phabricator.wikimedia.org/T132733#2274399 (10KartikMistry) [07:25:00] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#1177707 (10MoritzMuehlenhoff) @hashar: This bug can be closed, right? It seems T131749 fixed the remaining dependency problems. [07:54:27] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review, 15User-greg: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2274560 (10hashar) From Mukunda email to wikitech-l: > Starting Thursday May 12th, 13:00 PDT ( 20:00 GMT ) we will be having t... [08:19:01] (03PS3) 10Hashar: MobileFrontend and MultimediaViewer are moved from browsertests* to selenium* jobs. [integration/config] - 10https://gerrit.wikimedia.org/r/287195 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [08:20:08] (03CR) 10Hashar: [C: 032] MobileFrontend and MultimediaViewer are moved from browsertests* to selenium* jobs. [integration/config] - 10https://gerrit.wikimedia.org/r/287195 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [08:21:05] (03Merged) 10jenkins-bot: MobileFrontend and MultimediaViewer are moved from browsertests* to selenium* jobs. [integration/config] - 10https://gerrit.wikimedia.org/r/287195 (https://phabricator.wikimedia.org/T128190) (owner: 10Zfilipin) [08:24:52] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 06Multimedia, 10UploadWizard: Fix up the UploadWizard-api for beta cluster - https://phabricator.wikimedia.org/T134730#2274628 (10hashar) [08:25:05] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 06Multimedia, 10UploadWizard: Fix up the UploadWizard-api for beta cluster - https://phabricator.wikimedia.org/T134730#2274642 (10hashar) [08:25:09] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 06Multimedia, 10UploadWizard: Fix up the UploadWizard-api for beta cluster - https://phabricator.wikimedia.org/T134730#2274644 (10hashar) [08:25:23] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 06Multimedia, 10UploadWizard, 07WorkType-Maintenance: Fix up the UploadWizard-api for beta cluster - https://phabricator.wikimedia.org/T134730#2274628 (10hashar) p:05Triage>03Low [08:29:44] 10Beta-Cluster-Infrastructure, 06Services, 10cassandra: deployment-restbase01 puppet fails due to issues with cassandra package - https://phabricator.wikimedia.org/T134630#2274653 (10mobrovac) @Eevans let's remove the package hold on `deployment-restbase01` ? That forced upgrade is not a common scenario, so... [08:34:21] (03PS2) 10Hashar: [MultimediaPlayer] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287552 (owner: 10Paladox) [08:34:23] (03PS2) 10Hashar: [MultiUpload] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287554 (owner: 10Paladox) [08:34:25] (03PS2) 10Hashar: [MyVariables] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287557 (owner: 10Paladox) [08:34:27] (03PS2) 10Hashar: [MSSQLBackCompat] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287550 (owner: 10Paladox) [08:34:29] (03PS2) 10Hashar: [MsLinks] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287548 (owner: 10Paladox) [08:34:31] (03PS2) 10Hashar: [Mpdf] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287546 (owner: 10Paladox) [08:34:33] (03PS2) 10Hashar: [Moodle] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287544 (owner: 10Paladox) [09:11:19] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-login-and-signup, 10MediaWiki-extensions-CentralAuth, 07Beta-Cluster-reproducible: Getting logged out frequently in Beta Cluster - https://phabricator.wikimedia.org/T130896#2275212 (10MarcoAurelio) [09:51:56] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#2275316 (10hashar) We still ship `php5-fss` on Jessie (and Trusty) via `mediawiki::packages::php5`. Apparently it is not nee... [10:03:44] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: CI trusty slaves running out of memory - https://phabricator.wikimedia.org/T126545#2275331 (10hashar) [10:03:46] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, and 2 others: [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139#2275332 (10hashar) [10:03:48] 10Continuous-Integration-Infrastructure, 07Blocked-on-Operations, 13Patch-For-Review: Disable HHVM fcgi server on CI slaves - https://phabricator.wikimedia.org/T126594#2275327 (10hashar) 05Open>03Resolved a:03hashar Thank you @akosiaris . I have confirmed HHVM is gone from permanent CI slaves and from... [10:17:25] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #16: 04FAILURE in 1 min 58 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/16/ [10:18:01] 06Release-Engineering-Team, 13Patch-For-Review, 05Release: MW-1.27.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T131557#2275388 (10hashar) 1.27.0-wmf.23 got rollbacked to 1.27.0-wmf.22 due to a performance regression {T134553}. [10:42:23] 06Release-Engineering-Team, 05MW-1.27-release, 07Performance, 07Regression, 07User-notice: First paint time regression on Wikimedia page views with wmf.23 roll-out - https://phabricator.wikimedia.org/T134553#2269227 (10Gilles) I'm not sure that firstPaint has quite recovered since the revert. 7-day view... [10:59:58] 06Release-Engineering-Team, 05MW-1.27-release, 07Performance, 07Regression, 07User-notice: First paint time regression on Wikimedia page views with wmf.23 roll-out - https://phabricator.wikimedia.org/T134553#2275465 (10Gilles) While the most impressive week-over-week deviation was indeed while wmf.23 was... [11:14:02] 06Release-Engineering-Team, 05MW-1.27-release, 07Performance, 07Regression, 07User-notice: First paint time regression on Wikimedia page views with wmf.23 roll-out - https://phabricator.wikimedia.org/T134553#2275522 (10Gilles) I'm not sure if the sampling rate is sufficient, but it seems like mobile isn'... [11:18:14] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [12:42:45] (03PS3) 10Hashar: Change typos job to use extended regexp [integration/config] - 10https://gerrit.wikimedia.org/r/286937 (https://phabricator.wikimedia.org/T133047) [12:44:01] (03CR) 10Hashar: [C: 032] "Puppet change got merged. Job is refreshed." [integration/config] - 10https://gerrit.wikimedia.org/r/286937 (https://phabricator.wikimedia.org/T133047) (owner: 10Hashar) [12:45:18] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Write a test to check for clearly bogus hostnames - https://phabricator.wikimedia.org/T133047#2275809 (10hashar) One can add more extended regular expression to the `/typos` file. Eg the ones Faidon proposed such... [12:46:06] (03Merged) 10jenkins-bot: Change typos job to use extended regexp [integration/config] - 10https://gerrit.wikimedia.org/r/286937 (https://phabricator.wikimedia.org/T133047) (owner: 10Hashar) [13:12:52] 06Release-Engineering-Team, 05MW-1.27-release, 07Performance, 07Regression, 07User-notice: First paint time regression on Wikimedia page views with wmf.23 roll-out - https://phabricator.wikimedia.org/T134553#2275914 (10Gilles) Alright, I think I've found the culprit for wmf.23. This commit is part of the... [13:36:37] gilles: nice to you see looking at the firstPaint issue :)) [13:37:08] maybe you can backport Ori patches fixing the timing to .wmf.22 and see what happens [13:37:59] this way we can collect metric for a few hours and if all fine switch back to wmf.23 tonight [13:44:09] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2275973 (10faidon) I think having a flat service deployment model at the moment is fine, both be... [13:46:54] hashar: I'm not sure that we'll see much of an effect with that, considering that the effect seems to have been "sticky" to a degree [13:47:19] the fact that we're collecting a smaller share of broken values is enough of an explanation, whichever the cause may be [13:47:50] gilles: if we can get .23 tonight that would let us resume the mw cadence (i.e. switch to 1.28.0-wmf.1 tomorrow ) [13:48:48] right, I think we should just do that. the issue is clearly coming from the values collected from eventlogging, and nothing indicates that wmf.23 is problematic [13:49:31] resuming business as usual is more important than confirmation of the exact phenomenon at play. we shouldn't collect negative fistPaint values in the first place! [13:50:01] anyway, gtg, I'll be back in a couple of hours [13:52:48] 10Beta-Cluster-Infrastructure, 06Services, 10cassandra: deployment-restbase01 puppet fails due to issues with cassandra package - https://phabricator.wikimedia.org/T134630#2275994 (10Eevans) >>! In T134630#2274653, @mobrovac wrote: > @Eevans let's remove the package hold on `deployment-restbase01` ? That for... [14:00:55] 10Beta-Cluster-Infrastructure, 07Puppet, 07Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2276015 (10mobrovac) [14:00:57] 10Beta-Cluster-Infrastructure, 06Services, 10cassandra: deployment-restbase01 puppet fails due to issues with cassandra package - https://phabricator.wikimedia.org/T134630#2276012 (10mobrovac) 05Open>03Resolved a:03Eevans >>! In T134630#2275994, @Eevans wrote: > Ok, {{done}}. Thanks! Puppet now runs... [14:02:48] Yippee, build fixed! [14:02:48] Project selenium-Echo » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #17: 09FIXED in 49 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/17/ [14:02:57] Yippee, build fixed! [14:02:58] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #17: 09FIXED in 59 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/17/ [14:21:01] Yippee, build fixed! [14:21:02] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #5: 09FIXED in 18 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/5/ [14:23:51] PROBLEM - Puppet run on integration-slave-trusty-1015 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:25:07] Yippee, build fixed! [14:25:08] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #5: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/5/ [14:34:38] magic [14:34:47] zeljkof: some browser tests magically fixed :-} [14:35:05] hashar: I have rerun them :| [14:35:17] some jobs are just not stable :P [14:37:13] hashar: I am not sure what is the problem here [14:37:14] https://integration.wikimedia.org/ci/view/Selenium/job/selenium-MobileFrontend/5/testReport/ [14:37:27] who knows ? :-} [14:37:32] "0 failures" [14:37:38] but the firefox job is broken [14:37:50] https://integration.wikimedia.org/ci/view/Selenium/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/5/consoleFull [14:37:50] argh! [14:38:15] something in the stack (jenkins/cucumber...) is not catching that there was an error [14:38:26] 00:15:50.695 unexpected HTTP response (500) (MediawikiApi::HttpError) [14:38:28] 10Continuous-Integration-Infrastructure, 07WorkType-Maintenance: integration-lightslave-jessie-1002 went out of disk space - https://phabricator.wikimedia.org/T113474#2276219 (10hashar) Giving it a try again. Created integration-lightslave-jessie-1003 with 40GB disk space (instead of 20GB). [14:38:30] clearly failed [14:38:41] and the job is red [14:38:51] but then the report says "no failures" [14:39:00] * zeljkof is pulling his hair out [14:40:13] =o [14:40:33] https://integration.wikimedia.org/zuul/ just says status.json: [14:40:36] is stuff broken? :D [14:40:41] PROBLEM - Puppet run on integration-lightslave-jessie-1003 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [14:45:01] hashar: I think I know what the problem is [14:45:11] (re selenium tests) [14:45:31] the scenario failed during "background" so it was skipped, and officially did not fail [14:45:33] :| [14:45:49] hence failed job but no failed scenarios [14:50:22] 10Continuous-Integration-Infrastructure, 07Zuul: Zuul deadlock change_queue.isActionable(item) AttributeError: 'NoneType' object has no attribute 'isActionable' - https://phabricator.wikimedia.org/T134765#2276290 (10hashar) [14:51:25] hashar: did I break zuul? [14:52:17] I totally did :D [14:52:20] https://usercontent.irccloud-cdn.com/file/3AUXgNOK/ [14:52:34] hashar: Hi, thanks for merging my patches today :) [14:52:50] might be due to my use of depends on in https://gerrit.wikimedia.org/r/#/c/269467/5 which links to a changeid where i guess zuul doesnt know about the repo! [14:55:44] 10Continuous-Integration-Infrastructure, 07Zuul: Zuul deadlock change_queue.isActionable(item) AttributeError: 'NoneType' object has no attribute 'isActionable' - https://phabricator.wikimedia.org/T134765#2276326 (10hashar) Caused by https://gerrit.wikimedia.org/r/#/c/269467/ which requires a change on `analy... [14:56:05] 10Continuous-Integration-Infrastructure, 07Zuul: Zuul deadlock change_queue.isActionable(item) AttributeError: 'NoneType' object has no attribute 'isActionable' - https://phabricator.wikimedia.org/T134765#2276329 (10hashar) [14:56:12] 10Continuous-Integration-Infrastructure, 07Upstream, 07WorkType-Maintenance, 07Zuul: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2079136 (10hashar) [14:56:53] yup, my bad :P [14:58:14] 07Browser-Tests, 10Reading-Web-Backlog: Run browser tests on beta cluster from desktop domain - https://phabricator.wikimedia.org/T130429#2276336 (10dr0ptp4kt) [14:58:32] (03PS1) 10Hashar: [analytics/wmde/scripts] define noop [integration/config] - 10https://gerrit.wikimedia.org/r/287634 (https://phabricator.wikimedia.org/T128569) [14:59:19] (03CR) 10Hashar: [C: 032] [analytics/wmde/scripts] define noop [integration/config] - 10https://gerrit.wikimedia.org/r/287634 (https://phabricator.wikimedia.org/T128569) (owner: 10Hashar) [14:59:21] (03CR) 10jenkins-bot: [V: 04-1] [analytics/wmde/scripts] define noop [integration/config] - 10https://gerrit.wikimedia.org/r/287634 (https://phabricator.wikimedia.org/T128569) (owner: 10Hashar) [15:00:09] (03CR) 10jenkins-bot: [V: 04-1] [analytics/wmde/scripts] define noop [integration/config] - 10https://gerrit.wikimedia.org/r/287634 (https://phabricator.wikimedia.org/T128569) (owner: 10Hashar) [15:01:19] (03PS2) 10Hashar: [analytics/wmde/scripts] define noop [integration/config] - 10https://gerrit.wikimedia.org/r/287634 (https://phabricator.wikimedia.org/T128569) [15:02:38] (03CR) 10Hashar: [C: 032] [analytics/wmde/scripts] define noop [integration/config] - 10https://gerrit.wikimedia.org/r/287634 (https://phabricator.wikimedia.org/T128569) (owner: 10Hashar) [15:03:28] (03Merged) 10jenkins-bot: [analytics/wmde/scripts] define noop [integration/config] - 10https://gerrit.wikimedia.org/r/287634 (https://phabricator.wikimedia.org/T128569) (owner: 10Hashar) [15:03:31] :D [15:04:36] 10Continuous-Integration-Infrastructure, 07WorkType-Maintenance: integration-lightslave-jessie-1002 went out of disk space - https://phabricator.wikimedia.org/T113474#2276354 (10hashar) 05Open>03Resolved a:03hashar Replaced by integration-lightslave-jessie-1003 which has 40GBytes [15:06:04] RECOVERY - Puppet run on integration-lightslave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:38] PROBLEM - Host integration-lightslave-jessie-1002 is DOWN: CRITICAL - Host Unreachable (10.68.17.184) [15:08:31] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Tracking: Create CI slaves using Debian Jessie (tracking) - https://phabricator.wikimedia.org/T94836#2276391 (10faidon) [15:08:33] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#2276389 (10faidon) 05Open>03Resolved a:03faidon [15:12:10] Yippee, build fixed! [15:12:11] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #6: 09FIXED in 25 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/6/ [15:13:34] 10Continuous-Integration-Infrastructure, 06Operations, 13Patch-For-Review: Provide Jessie package to fullfil Mediawiki::Packages requirement - https://phabricator.wikimedia.org/T95002#2276419 (10hashar) Puppet looks fine on the CI slave integration-slave-jessie-1001 :-} [15:20:45] 06Release-Engineering-Team, 13Patch-For-Review, 05Release: MW-1.27.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T131557#2276461 (10greg) Per T134553#2275914 we're good to go with wmf.23 again. [15:21:26] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all debian-glue jobs to Jessie slaves - https://phabricator.wikimedia.org/T95545#2276481 (10hashar) [15:21:28] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Tracking: Create CI slaves using Debian Jessie (tracking) - https://phabricator.wikimedia.org/T94836#2276476 (10hashar) 05stalled>03Resolved a:03hashar Looks fine. integration-slave-jessie-1001 has been created a while ago and puppet is fine. [15:21:30] 10Continuous-Integration-Config, 13Patch-For-Review, 07WorkType-Maintenance: Create tox templates that are bound to jessie - https://phabricator.wikimedia.org/T103977#2276479 (10hashar) [15:21:32] 10Continuous-Integration-Config, 10Labs-project-extdist, 13Patch-For-Review, 07WorkType-Maintenance: Move labs/tools/extdist tox tests to jessie - https://phabricator.wikimedia.org/T102894#2276480 (10hashar) [15:22:04] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #7: 04FAILURE in 4 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/7/ [15:26:28] !log Creating integration-slave-jessie-1001 T95545 [15:26:29] T95545: Migrate all debian-glue jobs to Jessie slaves - https://phabricator.wikimedia.org/T95545 [15:26:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:30:07] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Ownership of Selenium tests - https://phabricator.wikimedia.org/T134492#2276548 (10greg) [15:32:21] 06Release-Engineering-Team, 15User-greg: Determine parental leave and delegation plan - https://phabricator.wikimedia.org/T131198#2276563 (10greg) 05Open>03Resolved State answered question(s). Submitted claim to CA EDD (disability department, where our Paid Family Leave comes from). WMF HR has the plan. As... [15:32:33] 06Release-Engineering-Team, 05MW-1.27-release, 07Performance, 07Regression, 07User-notice: First paint time regression on Wikimedia page views with wmf.23 roll-out - https://phabricator.wikimedia.org/T134553#2276566 (10Gilles) @ori by email: > > These values don't make it to Graphite (see NavTiming. P... [15:38:06] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 71.43% of data above the critical threshold [0.0] [15:38:14] Hey all — is it OK if I cause some downtime for deployment-fluorine ? [15:38:56] andrewbogott: sure. [15:39:00] thanks [15:39:03] !log Adding label contintLabsSlave to integration-slave-jessie1001 and integration-slave-jessie1002 [15:39:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:39:14] !log migrating deployment-flourine to labvirt1009 [15:39:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:41:58] 06Release-Engineering-Team, 13Patch-For-Review, 05Release: MW-1.27.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T131557#2276598 (10greg) maybe spoke too soon..... let's confirm with Gilles before moving forward [15:44:56] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Tracking: Create CI slaves using Debian Jessie (tracking) - https://phabricator.wikimedia.org/T94836#2276608 (10hashar) https://gerrit.wikimedia.org/r/287648 drops the transient `role::ci::slave::labs::light` which is no more needed. I have also... [15:46:23] PROBLEM - Host deployment-fluorine is DOWN: CRITICAL - Host Unreachable (10.68.16.198) [15:46:41] PROBLEM - Host integration-lightslave-jessie-1003 is DOWN: CRITICAL - Host Unreachable (10.68.21.134) [15:47:17] 06Release-Engineering-Team, 05MW-1.27-release, 07Performance, 07Regression, 07User-notice: First paint time regression on Wikimedia page views with wmf.23 roll-out - https://phabricator.wikimedia.org/T134553#2276613 (10Gilles) To which I replied: > I see the frontend.navtiming prefix in there. Do the co... [15:54:31] (03PS1) 10Hashar: Migrate all debian-glue jobs to Jessie slaves [integration/config] - 10https://gerrit.wikimedia.org/r/287649 [15:54:35] so, anybody with that mythical thing called spare time to deploy the NavigationTiming backport for me? I'll prepare the changesets now [15:54:36] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Automate the generation deployment keys (keyholder-managed ssh keys) - https://phabricator.wikimedia.org/T133211#2276655 (10mmodell) @faidon: Yes please join the meeting. [15:56:04] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all debian-glue jobs to Jessie slaves - https://phabricator.wikimedia.org/T95545#2276663 (10hashar) a:03hashar [15:56:36] twentyafterfour: deployment-tin refuses ssh connection again :(( [15:56:44] oh and good morning sorry [15:57:15] I could not found the task, just found out Jenkins can't add it back as a slave https://integration.wikimedia.org/ci/computer/deployment-tin.eqiad/log :( [15:58:01] hashar: looking into it [15:58:27] Project beta-code-update-eqiad build #103795: 04FAILURE in 3 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/103795/ [15:58:28] Project beta-update-databases-eqiad build #8446: 04FAILURE in 2.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/8446/ [15:59:10] 10Beta-Cluster-Infrastructure: deployment-tin ssh: Connection closed by UNKNOWN - https://phabricator.wikimedia.org/T134777#2276692 (10hashar) [15:59:14] filled at https://phabricator.wikimedia.org/T134777 [15:59:16] twentyafterfour: got it as https://phabricator.wikimedia.org/T134777 [15:59:33] twentyafterfour: probably not that urgent ;D that can wait after our weekly meeting I guess [16:00:57] 10Beta-Cluster-Infrastructure: deployment-tin ssh: Connection closed by UNKNOWN - https://phabricator.wikimedia.org/T134777#2276710 (10hashar) A line in pam access.conf has some extra whitespace. Maybe that is a cause? ``` root@deployment-salt:~ # salt -v 'deployment-tin*' cmd.run 'cat /etc/security/access.con... [16:02:00] 10Beta-Cluster-Infrastructure: deployment-tin ssh: Connection closed by UNKNOWN - https://phabricator.wikimedia.org/T134777#2276692 (10Krenair) This has been causing issues for me on and off for the past week (?) or so Works if you connect as root directly [16:02:09] twentyafterfour: it is back :) [16:03:19] I think there is a race condition in https://phabricator.wikimedia.org/rOPUP1b9c846dc69fecb113b6c91a4a9573d6e98636e8 [16:04:53] Yippee, build fixed! [16:04:54] Project beta-code-update-eqiad build #103796: 09FIXED in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/103796/ [16:06:24] Is there a way to easily see open patches against a repo in Phabricator? [16:06:28] Or a way to subscribe to a repo? [16:06:38] Yippee, build fixed! [16:06:39] Project beta-update-databases-eqiad build #8447: 09FIXED in 46 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/8447/ [16:06:46] twentyafterfour: all good :-} [16:07:06] Krinkle: herald might offer a way to subscribe to all changes to a given repo [16:07:42] gilles: My first question is mainly about transparency, insight and collaboration, not notification. One shoudl be able to see how a project is doing or if there are any pending patches. [16:07:56] It seems it is lacking any link between differential and diffusion in the interface to see open patches of that kind. [16:08:27] So unless someone involved you as reviewer, it's mostly invisible. [16:09:50] 10Deployment-Systems, 10MediaWiki-Configuration, 05MW-1.27-release-notes, 13Patch-For-Review: extension-list should live in the mediawiki branch rather than mediawiki-config - https://phabricator.wikimedia.org/T125678#2276740 (10mmodell) [16:12:31] 06Release-Engineering-Team, 13Patch-For-Review, 05Release: MW-1.27.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T131557#2276745 (10Gilles) @greg can someone from releng deploy this backport for me? https://gerrit.wikimedia.org/r/287654 [16:12:54] 07Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: [Task] enable rubocop in Wikibase - https://phabricator.wikimedia.org/T131139#2276749 (10JanZerebecki) [16:13:25] 06Release-Engineering-Team, 13Patch-For-Review, 05Release: MW-1.27.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T131557#2276753 (10hashar) Looks like we want to deploy 1.27.0-wmf.23 again this Monday. Make sure it is all happy and if tomorrow it still fine we can resume the regular deploy... [16:16:20] Yippee, build fixed! [16:16:21] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #8: 09FIXED in 26 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/8/ [16:19:26] (03PS1) 10JanZerebecki: wikidata/browsertests: add rake [integration/config] - 10https://gerrit.wikimedia.org/r/287655 (https://phabricator.wikimedia.org/T131139) [16:20:55] (03CR) 10Hashar: [C: 031] wikidata/browsertests: add rake [integration/config] - 10https://gerrit.wikimedia.org/r/287655 (https://phabricator.wikimedia.org/T131139) (owner: 10JanZerebecki) [16:21:12] (03CR) 10JanZerebecki: [C: 032] wikidata/browsertests: add rake [integration/config] - 10https://gerrit.wikimedia.org/r/287655 (https://phabricator.wikimedia.org/T131139) (owner: 10JanZerebecki) [16:21:58] (03Merged) 10jenkins-bot: wikidata/browsertests: add rake [integration/config] - 10https://gerrit.wikimedia.org/r/287655 (https://phabricator.wikimedia.org/T131139) (owner: 10JanZerebecki) [16:22:01] RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [16:23:19] RECOVERY - Host deployment-fluorine is UP: PING OK - Packet loss = 0%, RTA = 0.84 ms [16:32:42] !log reloading zuul for 3e2ab56..d663fd0 [16:32:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:03:05] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices, 13Patch-For-Review: Set up Code Review office hours - https://phabricator.wikimedia.org/T128371#2277050 (10greg) [17:05:16] gilles: ostriches is going to redeploy 1.27.0-wmf.23 tonight apparently [17:05:29] gilles: I guess he will come back to you / ori soonish [17:05:35] alright [17:06:31] I am off! [17:51:42] 06Release-Engineering-Team, 10scap, 03Scap3 (Scap3-MediaWiki-MVP): git/http operations in scap should be secure - https://phabricator.wikimedia.org/T127498#2045336 (10mmodell) We discussed this in the deployment meeting today. @faidon suggested one option is for scap to deploy ephemeral tls keys to the proxy... [18:05:38] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:10:37] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:44] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:47:30] PROBLEM - Puppet run on deployment-kafka02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:54:38] 10Continuous-Integration-Infrastructure, 07Puppet: Puppet fails on Jessie permanent CI slaves - https://phabricator.wikimedia.org/T134808#2277778 (10hashar) [20:02:29] RECOVERY - Puppet run on deployment-kafka02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:04:01] !log CI, removing ganglia configuration entirely via: salt -v '*' cmd.run 'dpkg --purge ganglia-monitor' # T134808 [20:04:02] T134808: Puppet fails on Jessie permanent CI slaves - https://phabricator.wikimedia.org/T134808 [20:04:05] which is going to mass fail puppet [20:04:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:06:27] !log CI, removing ganglia configuration entirely via: salt -v '*' cmd.run 'rm -fRv /etc/ganglia' # T134808 [20:06:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:07:03] T134808: Puppet fails on Jessie permanent CI slaves - https://phabricator.wikimedia.org/T134808 [20:12:50] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:13:45] !log beta: salt -v '*' cmd.run 'dpkg --purge libganglia1 ganglia-monitor; rm -fR /etc/ganglia' # T134808 [20:13:46] T134808: Puppet fails on Jessie permanent CI slaves - https://phabricator.wikimedia.org/T134808 [20:13:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:14:18] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Puppet: Puppet fails on labs instances using apache::site puppet class - https://phabricator.wikimedia.org/T134808#2277814 (10hashar) [20:14:34] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Puppet: Puppet fails on labs instances using apache::site puppet class - https://phabricator.wikimedia.org/T134808#2277778 (10hashar) p:05Triage>03Normal a:03hashar [20:17:39] PROBLEM - Host deployment-kafka02 is DOWN: CRITICAL - Host Unreachable (10.68.17.156) [20:17:39] 10Deployment-Systems, 06Release-Engineering-Team, 06Operations, 06Services: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#2277846 (10thcipriani) We discussed this ticket WRT to scap3 work briefly in the deployment working group meeting today (https://w... [20:19:30] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Puppet: Puppet fails on labs instances using apache::site puppet class - https://phabricator.wikimedia.org/T134808#2277778 (10hashar) [20:27:06] PROBLEM - Puppet run on deployment-pdf02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:31:29] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Puppet: Puppet fails on labs instances due to Ganglia (ex: using apache::site puppet class) - https://phabricator.wikimedia.org/T134808#2277894 (10hashar) [20:36:15] 10Beta-Cluster-Infrastructure, 07Puppet, 07Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2277910 (10hashar) [20:36:17] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Puppet: Puppet fails on labs instances due to Ganglia (ex: using apache::site puppet class) - https://phabricator.wikimedia.org/T134808#2277911 (10hashar) [20:37:07] 10Beta-Cluster-Infrastructure, 07Puppet, 07Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2192864 (10hashar) Puppet failures related to Ganglia would be due to T134808 which have fixed cherry picked on beta puppet master but do not cover every cases. [20:39:04] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Puppet: Puppet fails on labs instances due to Ganglia (ex: using apache::site puppet class) - https://phabricator.wikimedia.org/T134808#2277778 (10Dzahn) @hashar re: ganglia on labs, also see T115330. ganglia-moni... [20:50:22] Error: Could not set 'file' on ensure: No such file or directory - /etc/ganglia/conf.d/ocg.pyconf20160509-10639-p2bf0l.lock at 20:/etc/puppet/modules/ocg/manifests/ganglia.pp [20:50:22] Error: Could not set 'file' on ensure: No such file or directory - /etc/ganglia/conf.d/ocg.pyconf20160509-10639-p2bf0l.lock at 20:/etc/puppet/modules/ocg/manifests/ganglia.pp [20:50:22] Wrapped exception: [20:50:22] No such file or directory - /etc/ganglia/conf.d/ocg.pyconf20160509-10639-p2bf0l.lock [20:50:23] Error: /Stage[main]/Ocg::Ganglia/File[/etc/ganglia/conf.d/ocg.pyconf]/ensure: change from absent to file failed: Could not set 'file' on ensure: No such file or directory - /etc/ganglia/conf.d/ocg.pyconf20160509-10639-p2bf0l.lock at 20:/etc/puppet/modules/ocg/manifests/ganglia.pp [20:50:27] do we know about this one? [20:50:33] hashar, ^ [20:50:59] see the backlog ? :-} [20:51:11] Krenair: https://phabricator.wikimedia.org/T134808 [20:51:50] on -pdf? [20:51:57] does that use apache::site? [20:53:56] GANGLIA [20:54:04] apache::site --> apache::monitoring ---> ganglia [20:54:20] so that instance has whatever other code path which ends up with --> ganglia [20:54:33] or attempts to create a file under /etc/ganglia which I have removed entirely [20:54:46] (sorry for capslock) [20:57:59] !log Unbroke puppet on integration-raita.integration.eqiad.wmflabs . Puppet was blocked because role::ci::raita was no more. Fixed by rebasing https://gerrit.wikimedia.org/r/#/c/208024 T115330 [20:58:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:58:10] T115330: block labs IPs from sending data to prod ganglia - https://phabricator.wikimedia.org/T115330 [20:59:22] Krenair: manifests/role/ocg.pp: include ::ocg::ganglia [20:59:22] :D [21:00:12] I fixed it on apache https://gerrit.wikimedia.org/r/#/c/287695/1/modules/apache/manifests/monitoring.pp,cm [21:00:16] similar thing for hhvm [21:00:54] Oh Isee [21:00:56] I see* [21:07:06] some others will fail as well [21:07:10] most probably [21:07:17] but that will be for another time *escapes* [21:12:59] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all debian-glue jobs to Jessie slaves - https://phabricator.wikimedia.org/T95545#2278068 (10hashar) The Jessie permanent slaves integration-slave-jessie-1001 and integration-slave-jessie-1002 are passing puppet and have the same role::ci::lab... [21:13:33] (03PS2) 10Hashar: Migrate all debian-glue jobs to Jessie slaves [integration/config] - 10https://gerrit.wikimedia.org/r/287649 (https://phabricator.wikimedia.org/T95545) [21:13:58] (03CR) 10Hashar: [C: 04-1] "Not ready, gotta verify the jobs pass." [integration/config] - 10https://gerrit.wikimedia.org/r/287649 (https://phabricator.wikimedia.org/T95545) (owner: 10Hashar) [21:20:34] (03PS1) 10Hashar: typos job now excludes .git directories [integration/config] - 10https://gerrit.wikimedia.org/r/287755 (https://phabricator.wikimedia.org/T133047) [21:28:19] 06Release-Engineering-Team, 05MW-1.27-release, 07Performance, 07Regression, 07User-notice: First paint time regression on Wikimedia page views with wmf.23 roll-out - https://phabricator.wikimedia.org/T134553#2278207 (10Gilles) Theory confirmed by backporting the change to wmf22: {F3988494} The gap that... [21:36:16] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Operations: Write a test to check for clearly bogus hostnames - https://phabricator.wikimedia.org/T133047#2278260 (10hashar) operations/puppet.git has some harness now. For mediawiki-config either we duplicate the typo file or I can craft a job... [21:41:12] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Operations: Write a test to check for clearly bogus hostnames - https://phabricator.wikimedia.org/T133047#2278296 (10demon) >>! In T133047#2278260, @hashar wrote: > operations/puppet.git has some harness now. > > For mediawiki-config either we d... [21:48:50] (03PS1) 10Paladox: [NamespaceHTML] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287762 [21:54:21] (03PS1) 10Paladox: [NamespacePaths] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287765 [21:59:03] (03PS1) 10Paladox: [NamespaceRelations] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287767 [22:14:13] (03PS1) 10Paladox: [Negref] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287770 [22:18:46] (03PS1) 10Paladox: [NetworkAuth] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287773 [22:19:07] (03CR) 10Paladox: "check experimental" [integration/config] - 10https://gerrit.wikimedia.org/r/287773 (owner: 10Paladox) [22:25:24] (03PS1) 10Paladox: [News] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287775 [22:30:59] (03PS1) 10Paladox: [NewsBox] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287779 [22:37:02] (03PS1) 10Paladox: [NewSignupPage] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287781 [22:43:17] (03PS1) 10Paladox: [NewUsersList] Add npmt test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287784 [22:49:07] (03PS1) 10Paladox: [NoTitle] Add npm test and composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/287787 [23:22:16] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 3.06 ms [23:28:17] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [23:29:51] 06Release-Engineering-Team, 15User-greg: Plan 2016 RelEng team offsite - https://phabricator.wikimedia.org/T134830#2278940 (10greg) [23:57:07] 10Deployment-Systems, 06Release-Engineering-Team, 03Scap3, 06Operations: setup automatic deletion of old l10nupdate - https://phabricator.wikimedia.org/T130317#2132321 (10greg) >>! In T130317#2263058, @ori wrote: > @mmodell, blocking this on porting l10nupdate to scap doesn't seem reasonable. Could you sim... [23:58:17] 10Deployment-Systems, 06Release-Engineering-Team, 03Scap3, 06Operations: setup automatic deletion of old l10nupdate - https://phabricator.wikimedia.org/T130317#2279063 (10mmodell) @greg: sure @ori: sorry I missed this before.