[00:19:42] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 1.71 ms [00:24:43] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [00:27:14] Yippee, build fixed! [00:27:15] Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #15: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/15/ [00:28:18] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 1.03 ms [00:58:18] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [01:12:17] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 0.73 ms [01:19:55] 10Beta-Cluster-Infrastructure, 10Flow, 03Collab-Team-2016-Apr-Jun-Q4: Set up second External Store cluster on Beta - https://phabricator.wikimedia.org/T128417#2283665 (10Mattflaschen) a:03Mattflaschen [01:34:54] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #622: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/622/ [01:39:09] 10Beta-Cluster-Infrastructure, 10Flow, 03Collab-Team-2016-Apr-Jun-Q4: Set up second External Store cluster on Beta - https://phabricator.wikimedia.org/T128417#2283675 (10Mattflaschen) [01:42:54] 10Beta-Cluster-Infrastructure, 10Flow, 03Collab-Team-2016-Apr-Jun-Q4: Set up second External Store cluster on Beta - https://phabricator.wikimedia.org/T128417#2283677 (10Mattflaschen) Created second one (I used a less clear name the first time so I had to fix and re-run): ``` export MEDIAWIKI_STAGING_DIR; .... [01:44:18] !log Created Flow-specific External Store tables (blobs_flow1) on all wiki databases on Beta Cluster: T128417 [01:44:19] T128417: Set up second External Store cluster on Beta - https://phabricator.wikimedia.org/T128417 [01:44:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:47:20] 10Beta-Cluster-Infrastructure, 10Flow, 03Collab-Team-2016-Apr-Jun-Q4: Set up Flow-specific External Store cluster on Beta (secondary to the main one) - https://phabricator.wikimedia.org/T128417#2283680 (10Mattflaschen) [02:38:20] PROBLEM - Parsoid on deployment-parsoid06 is CRITICAL: Connection refused [02:39:18] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [03:18:29] 10scap, 10ContentTranslation-Deployments, 10ContentTranslation-cxserver, 10MediaWiki-extensions-ContentTranslation, and 3 others: Deploy CXServer with scap3 - https://phabricator.wikimedia.org/T120104#2283754 (10KartikMistry) https://gerrit.wikimedia.org/r/#/c/286395/ is scheduled to deploy today along wit... [03:58:06] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [03:59:12] PROBLEM - Puppet run on phab-beta is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:38:04] RECOVERY - Puppet run on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:18:03] 10Beta-Cluster-Infrastructure: Migrate beta cluster memcached from Precise to Jessie - https://phabricator.wikimedia.org/T134974#2284125 (10hashar) [08:20:38] !log Creating deployment-memc04 and deployment-memc05 to switch beta cluster memcached to Jessie. m1.medium with security policy "cache" T13497 [08:20:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:20:48] T13497: Component for Wiktionary within Bugzilla - https://phabricator.wikimedia.org/T13497 [08:32:31] 10Beta-Cluster-Infrastructure: Migrate beta cluster memcached from Precise to Jessie - https://phabricator.wikimedia.org/T134974#2284157 (10hashar) a:03hashar [08:38:51] PROBLEM - Puppet run on deployment-memc05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:42:08] 10Beta-Cluster-Infrastructure: Migrate beta cluster memcached from Precise to Jessie - https://phabricator.wikimedia.org/T134974#2284167 (10hashar) New instances: | Hostname | IP address |--|-- | deployment-memc04 | 10.68.23.25 | deployment-memc05 | 10.68.23.49 [08:43:02] !log Beta: switching memcached to new Jessie servers by cherry picking https://gerrit.wikimedia.org/r/#/c/288156/ and running puppet on mw app servers #T134974 [08:43:04] T134974: Migrate beta cluster memcached from Precise to Jessie - https://phabricator.wikimedia.org/T134974 [08:43:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:43:45] RECOVERY - Puppet run on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [08:45:05] PROBLEM - Puppet run on deployment-memc04 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [08:46:52] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [08:49:38] !log Deleting instances deployment-memc02 and deployment-memc03 (Precise instances, migrated to Jessie) #T134974 [08:49:39] T134974: Migrate beta cluster memcached from Precise to Jessie - https://phabricator.wikimedia.org/T134974 [08:49:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:50:06] RECOVERY - Puppet run on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [08:50:30] PROBLEM - Puppet run on deployment-jobrunner01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:50:38] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Migrate beta cluster memcached from Precise to Jessie - https://phabricator.wikimedia.org/T134974#2284181 (10hashar) 05Open>03Resolved The Precise instances were the oldest on beta cluster created in February 2014 :-} [08:51:38] PROBLEM - Host deployment-memc03 is DOWN: CRITICAL - Host Unreachable (10.68.16.15) [08:53:34] PROBLEM - Host deployment-memc02 is DOWN: CRITICAL - Host Unreachable (10.68.16.14) [08:55:20] ^^^ I have killed memc02 and memc03 [08:57:15] 06Release-Engineering-Team, 10MediaWiki-extensions-General-or-Unknown, 07Wikimedia-log-errors: SpecialRecentChangesLinked::doMainQuery blocking database infrastructure - https://phabricator.wikimedia.org/T134976#2284195 (10jcrespo) [09:02:48] 06Release-Engineering-Team, 10MediaWiki-extensions-General-or-Unknown, 07Wikimedia-log-errors: SpecialRecentChangesLinked::doMainQuery blocking database infrastructure - https://phabricator.wikimedia.org/T134976#2284215 (10hashar) [09:02:50] 06Release-Engineering-Team, 05Release: 1.28.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T134249#2284214 (10hashar) [09:04:07] 06Release-Engineering-Team, 05Release: 1.28.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T134249#2259711 (10hashar) p:05Triage>03Normal a:03demon Done by @demon / @hashar I have added {T134249} as a blocker, some bot cause high usage database queries and we probably want to properly... [09:10:37] RECOVERY - Puppet run on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:10:42] 06Release-Engineering-Team, 10MediaWiki-extensions-General-or-Unknown, 07Wikimedia-log-errors: SpecialRecentChangesLinked::doMainQuery blocking database infrastructure - https://phabricator.wikimedia.org/T134976#2284195 (10hashar) I have made this task a blocker of {T134249} We had 1.27.0-wmf.23 rollbacked... [09:12:03] 06Release-Engineering-Team, 10MediaWiki-extensions-General-or-Unknown, 07Wikimedia-log-errors: SpecialRecentChangesLinked::doMainQuery blocking database infrastructure - https://phabricator.wikimedia.org/T134976#2284230 (10hashar) [09:32:27] 06Release-Engineering-Team, 10MediaWiki-extensions-General-or-Unknown, 07Wikimedia-log-errors: SpecialRecentChangesLinked::doMainQuery blocking database infrastructure - https://phabricator.wikimedia.org/T134976#2284255 (10hashar) Some more events a bit earlier https://logstash.wikimedia.org/#dashboard/temp/... [10:09:49] hashar: Hi could you review https://gerrit.wikimedia.org/r/#/c/288128/ please. [10:10:00] Im not sure if there is a cleaner way to do it. [10:12:19] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 0.73 ms [10:29:18] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [10:34:36] 06Release-Engineering-Team, 10MediaWiki-Special-pages, 07Wikimedia-log-errors: SpecialRecentChangesLinked::doMainQuery blocking database infrastructure - https://phabricator.wikimedia.org/T134976#2284377 (10Danny_B) [10:46:07] !log beta/ci puppetmaster : deleting old tags in /var/lib/git/operations/puppet and repacking the repos [10:46:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:52:47] (03PS1) 10Hashar: [operations/software] noop jobs [integration/config] - 10https://gerrit.wikimedia.org/r/288170 [10:55:20] (03CR) 10Hashar: [C: 032] [operations/software] noop jobs [integration/config] - 10https://gerrit.wikimedia.org/r/288170 (owner: 10Hashar) [10:56:13] (03Merged) 10jenkins-bot: [operations/software] noop jobs [integration/config] - 10https://gerrit.wikimedia.org/r/288170 (owner: 10Hashar) [11:42:23] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-aqs01.deployment-prep.eqiad.wmflabs doesn't respond to ssh / hung process - https://phabricator.wikimedia.org/T134981#2284433 (10hashar) [11:42:49] !log rebooting deployment-aqs01 via wikitech T134981 [11:42:50] T134981: deployment-aqs01.deployment-prep.eqiad.wmflabs doesn't respond to ssh / hung process - https://phabricator.wikimedia.org/T134981 [11:42:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:46:12] RECOVERY - Free space - all mounts on deployment-sentry2 is OK: OK: All targets OK [11:46:13] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-aqs01.deployment-prep.eqiad.wmflabs doesn't respond to ssh / hung process - https://phabricator.wikimedia.org/T134981#2284441 (10hashar) It is back. Puppet is lagged out: The last Puppet run was at Sat May 7 04:51:29 UTC 2016 (6174 minutes ago). [11:48:11] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-aqs01.deployment-prep.eqiad.wmflabs doesn't respond to ssh / hung process - https://phabricator.wikimedia.org/T134981#2284451 (10hashar) 05Open>03Resolved a:03hashar Puppet log that auto started on instance boot: ``` Notice: /Stage[main]/Scap/Package... [12:08:30] 10MediaWiki-Codesniffer: Require spaces in short array syntax - https://phabricator.wikimedia.org/T134982#2284482 (10gabriel-wmde) [12:22:59] Yippee, build fixed! [12:22:59] Project selenium-GettingStarted » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #16: 09FIXED in 58 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/16/ [12:34:49] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 1.20 ms [12:38:15] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [12:44:40] RECOVERY - Host integration-dev is UP: PING OK - Packet loss = 0%, RTA = 2.08 ms [12:49:40] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [12:51:42] !log creating integration-dev instance to hopefully have Shinken clean itself [12:51:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:52:42] !log deleted integration-dev [12:52:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:59:33] !log Dropping texlive and its dependencies from gallium. [12:59:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:04:24] Project selenium-Math » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #15: 04FAILURE in 24 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/15/ [13:04:27] Project selenium-Math » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #15: 04FAILURE in 27 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/15/ [13:17:19] (03PS1) 10Hashar: dib: add contint::packages::php [integration/config] - 10https://gerrit.wikimedia.org/r/288187 (https://phabricator.wikimedia.org/T119139) [13:18:43] (03CR) 10Hashar: [C: 032] dib: add contint::packages::php [integration/config] - 10https://gerrit.wikimedia.org/r/288187 (https://phabricator.wikimedia.org/T119139) (owner: 10Hashar) [13:19:31] RECOVERY - Keyholder status on mira is OK: OK: Less than 100.00% above the threshold [0.0] [13:19:40] twentyafterfour: The favicon of our phabricator is gone :-/ [13:21:36] (03Merged) 10jenkins-bot: dib: add contint::packages::php [integration/config] - 10https://gerrit.wikimedia.org/r/288187 (https://phabricator.wikimedia.org/T119139) (owner: 10Hashar) [13:33:00] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Ownership of Selenium tests - https://phabricator.wikimedia.org/T134492#2285063 (10JanZerebecki) [13:33:52] !log Added contint::packages::php to Nodepool images T119139 [13:33:53] T119139: [keyresult] Migrate php (Zend and HHVM) CI jobs to Nodepool - https://phabricator.wikimedia.org/T119139 [13:33:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:17:18] (03PS1) 10Hashar: dib: misc packages for MediaWiki testing [integration/config] - 10https://gerrit.wikimedia.org/r/288203 (https://phabricator.wikimedia.org/T119139) [14:18:25] (03CR) 10Hashar: [C: 032] dib: misc packages for MediaWiki testing [integration/config] - 10https://gerrit.wikimedia.org/r/288203 (https://phabricator.wikimedia.org/T119139) (owner: 10Hashar) [14:19:14] (03Merged) 10jenkins-bot: dib: misc packages for MediaWiki testing [integration/config] - 10https://gerrit.wikimedia.org/r/288203 (https://phabricator.wikimedia.org/T119139) (owner: 10Hashar) [14:36:45] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, 07WorkType-NewFunctionality: Migrate PHPUnit MediaWiki core jobs to Nodepool - https://phabricator.wikimedia.org/T135001#2285338 (10hashar) [14:37:39] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, 07WorkType-NewFunctionality: Migrate PHPUnit MediaWiki core jobs to Nodepool - https://phabricator.wikimedia.org/T135001#2285354 (10hashar) [14:39:24] (03PS2) 10Hashar: (WIP) Mediawiki PHPUnit to Nodepool (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/286497 (https://phabricator.wikimedia.org/T135001) [14:39:49] (03CR) 10jenkins-bot: [V: 04-1] (WIP) Mediawiki PHPUnit to Nodepool (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/286497 (https://phabricator.wikimedia.org/T135001) (owner: 10Hashar) [14:39:57] (03CR) 10Hashar: "I have filled T135001 to track this migration." [integration/config] - 10https://gerrit.wikimedia.org/r/286497 (https://phabricator.wikimedia.org/T135001) (owner: 10Hashar) [15:10:15] (03PS3) 10Hashar: (WIP) Mediawiki PHPUnit to Nodepool (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/286497 (https://phabricator.wikimedia.org/T135001) [15:10:46] (03CR) 10Hashar: "Rebased. fixed conflict in zuul/layout.yaml due to mediawiki-core-phpcs-trusty" [integration/config] - 10https://gerrit.wikimedia.org/r/286497 (https://phabricator.wikimedia.org/T135001) (owner: 10Hashar) [15:22:48] (03CR) 10Hashar: "Have split parsertests to their own jobs (more code duplication sadly)" [integration/config] - 10https://gerrit.wikimedia.org/r/286497 (https://phabricator.wikimedia.org/T135001) (owner: 10Hashar) [15:23:44] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, and 2 others: Migrate PHPUnit MediaWiki core jobs to Nodepool - https://phabricator.wikimedia.org/T135001#2285456 (10hashar) Triggering jobs manually from gallium.wikimedia.org:/home/hashar with: ```... [15:26:16] (03CR) 10Hashar: [C: 04-1] "Tidy is still not found :(" [integration/config] - 10https://gerrit.wikimedia.org/r/286497 (https://phabricator.wikimedia.org/T135001) (owner: 10Hashar) [15:29:23] 07Browser-Tests, 10Wikidata, 07Tracking: [tracking] make Wikidata browsertests non-flaky - https://phabricator.wikimedia.org/T92619#2285463 (10Jonas) [16:01:43] twentyafterfour: I think it is better if we don't make the code review office hours a recurring event: If you join a recurring event it doesn't allow you to join only for example specific weeks, phabricator shows that like you joined every week [16:03:54] 10releng-201516-q2, 10releng-201516-q3, 10scap, 03Scap3 (Scap3-Adoption-Phase1): [keyresult] Migrate all Service team owned services and MW to scap - https://phabricator.wikimedia.org/T109926#2285546 (10KartikMistry) [16:03:56] 10scap, 10ContentTranslation-Deployments, 10ContentTranslation-cxserver, 10MediaWiki-extensions-ContentTranslation, and 3 others: Deploy CXServer with scap3 - https://phabricator.wikimedia.org/T120104#2285545 (10KartikMistry) 05Open>03Resolved [16:04:05] Luke081515: hmm... [16:04:37] so make a new event for every week? Seems like a lot of extra work [16:05:55] twentyafterfour: or inform upstream about that, and try to find a solution ;) [16:06:33] see also: https://phabricator.wikimedia.org/T1035#2266735 [16:11:40] yay, cxserver is on scap3 [16:15:57] \O/ [16:16:00] kart_: congratulations :) [16:16:04] greg-g: ah, you're back? :D [16:16:34] Luke081515: ish :) 50% time until July 7th(ish) [16:16:49] ah, ok [16:17:06] working most days (except Fridays, generally) [16:19:48] 06Release-Engineering-Team, 15User-greg: Setup meeting with $people to discuss code hosting exception policy - https://phabricator.wikimedia.org/T109657#2285584 (10greg) p:05Normal>03Low [16:19:57] 06Release-Engineering-Team, 15User-greg: Publish WMF code-hosting exception policy - https://phabricator.wikimedia.org/T109919#2285585 (10greg) p:05Normal>03Low [16:21:20] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, and 2 others: Migrate PHPUnit MediaWiki core jobs to Nodepool - https://phabricator.wikimedia.org/T135001#2285587 (10hashar) PHPUnit without parser tests: | Job | Status |--|-- | mediawiki-phpunit-ph... [16:23:49] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 03releng-201516-q4, and 2 others: Migrate PHPUnit MediaWiki core jobs to Nodepool - https://phabricator.wikimedia.org/T135001#2285622 (10hashar) Trusty images have xhprof: ``` php5-xhprof: Installed: 0.9.4-1build1 Ca... [16:35:32] 07Browser-Tests, 10Wikidata, 07Tracking: [tracking] make Wikidata browsertests non-flaky - https://phabricator.wikimedia.org/T92619#2285689 (10JanZerebecki) [16:42:20] 10Continuous-Integration-Config, 10Analytics: Add a maven-release user to Gerrit {hawk} - https://phabricator.wikimedia.org/T132176#2285719 (10madhuvishy) [16:43:46] !log Reduced number of executors on Trusty instances from 3 to 2. Memory get exhausted causing the tmpfs to drop files and thus MW jobs to fail randomly. [16:43:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:45:42] 10Continuous-Integration-Config, 10Analytics: Add a maven-release user to Gerrit {hawk} - https://phabricator.wikimedia.org/T132176#2190712 (10madhuvishy) Can someone from releng help with this? Pinging @demon - It was mentioned to me sometime that he usually handles similar requests. This task involves making... [16:47:01] ostriches: ^ I can handle that for you if I have enough privs in gerrit... [16:47:29] (03CR) 1020after4: [C: 031] Add partial support for maven-release-plugin [integration/jenkins-job-builder] - 10https://gerrit.wikimedia.org/r/286788 (https://phabricator.wikimedia.org/T132175) (owner: 10Madhuvishy) [16:47:37] twentyafterfour: Anyone can make the user, there's no reason for me to do it. [16:47:53] I can always adjust ACLs later if someone doesn't have privs to do that on their repo (but they should....) [16:49:26] (03CR) 1020after4: (WIP) Zuul deployment with scap? (WIP) (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/286207 (https://phabricator.wikimedia.org/T129357) (owner: 10Hashar) [16:51:39] ostriches: twentyafterfour I dont mind making it - not sure about keeping the private key with myself though - wasn't sure if there was a good place to do that [16:52:36] twentyafterfour: I can make some tabs on the left in differential that show open patches. All of them not just user specific or certain ones [16:52:42] Mirrors gerrit open tab. [16:52:59] Not mirrors but does what gerrit open tab does [16:54:49] madhuvishy: No need to keep it yourself I suppose. Once it's stashed in jenkins you could trash it? [16:54:56] We could always re-gen later if it was lost. [16:55:01] ostriches: sure - that works [16:56:55] PROBLEM - SSH on integration-slave-trusty-1016 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:50] twentyafterfour: Please take a look at http://www.test-random-wikisaur.tk/differential/query/open/ [17:01:03] Now i have managed open, merged and abadoned [17:01:16] Should make viewing open patches more easy now. [17:02:12] 10MediaWiki-Codesniffer: Require spaces in short array syntax - https://phabricator.wikimedia.org/T134982#2284482 (10Legoktm) What version of MW-CS are you using? It should be enforced in 0.7.1: {0a0758b102797fd57fb88d3e450cb229723a8aa1} [17:02:46] ostriches do you like these tabs http://www.test-random-wikisaur.tk/differential/query/open/ to be in differential [17:02:52] Makes it easy now [17:02:55] 10Continuous-Integration-Config, 10Analytics: Add a maven-release user to Gerrit {hawk} - https://phabricator.wikimedia.org/T132176#2285850 (10madhuvishy) From chatting on irc, It looks like I can make it myself. So i will! Thanks y'all! [17:03:25] paladox: Considering I never use those tabs in Gerrit, *shrug* [17:03:36] I'm not opposed to adding them, just don't remove the existing ones. [17:04:17] ostriches: Oh, ok. I woulden remove any other tabs just add's the ones that gerrit uses. [17:11:50] (03CR) 10Madhuvishy: "This is being reviewed upstream here - https://review.openstack.org/#/c/313196/" [integration/jenkins-job-builder] - 10https://gerrit.wikimedia.org/r/286788 (https://phabricator.wikimedia.org/T132175) (owner: 10Madhuvishy) [17:14:12] ostriches and twentyafterfour: https://phabricator.wikimedia.org/D231 [18:40:16] 10Continuous-Integration-Config, 06Front-end-Standards-Group: Devise a recommended grunt configuration for linting and style-checking CSS files that isn't CSSlint - https://phabricator.wikimedia.org/T130721#2286268 (10Jdforrester-WMF) a:03Jdforrester-WMF [18:49:29] ostriches: what email address should I use for the release user account though? [18:50:49] Wherever you want the e-mails to go to. [18:53:57] ostriches: uhhh - I'd prefer if it was something generic - and not mine - mine is the only one I can confirm the account from [18:54:47] i guess i don't know how to get an email address for this purpose [19:01:49] OIT can make aliases :) [19:02:05] ostriches: cool i'll poke them [19:02:38] Can always swap the e-mail addy later too if we need to move forward sooner than we can get an alias. [19:10:13] ostriches: I think we can wait till they make an alias [19:10:17] thank you [19:12:03] yw [19:29:07] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 10Ops-Access-Requests: Allow RelEng nova log access - https://phabricator.wikimedia.org/T133992#2286452 (10chasemp) >>! In T133992#2282661, @hashar wrote: > We can revisit the list of contint-admins. I am not sure whether b... [19:54:58] 06Release-Engineering-Team, 06Operations, 10Wikimedia-General-or-Unknown: Inconsistently unable to download https://releases.wikimedia.org/mediawiki/1.26/mediawiki-1.26.2.tar.gz (returns zero-byte response) - https://phabricator.wikimedia.org/T135038#2286540 (10matmarex) [20:00:15] Krinkle and jdlrobson and James_F: Could we create a new repo to start the coversion of the wikimedia preset to eslint [20:00:16] See https://github.com/ntwb/eslint-config-wordpress [20:00:22] But not use it. [20:00:36] Until we start converting repos to it. [20:00:40] Please [20:00:42] paladox: Not yet. [20:01:00] We'll wait a month or two to see how it settles down first [20:01:03] James_F: Ok, we can use this tool https://github.com/brenolf/polyjuice [20:01:05] Ok [20:01:05] 06Release-Engineering-Team, 06Operations, 10Wikimedia-General-or-Unknown: Inconsistently unable to download https://releases.wikimedia.org/mediawiki/1.26/mediawiki-1.26.2.tar.gz (returns zero-byte response) - https://phabricator.wikimedia.org/T135038#2286576 (10hashar) [20:01:12] JSCS is still supported by upstream and not going away anytime soon. [20:01:24] Krinkle they said three months of updates [20:01:24] There is no gain in using eslint right now. [20:03:00] 06Release-Engineering-Team, 06Operations, 10Traffic, 10Wikimedia-General-or-Unknown: Inconsistently unable to download https://releases.wikimedia.org/mediawiki/1.26/mediawiki-1.26.2.tar.gz (returns zero-byte response) - https://phabricator.wikimedia.org/T135038#2286540 (10hashar) #traffic people would sure... [20:03:01] Per [20:03:03] In the end of the beginning [20:03:03] As mentioned earlier, we will continue to support JSCS for the next three months, fixing significant bugs. The JSCS repository will be left in place, so you are free to fork it and otherwise use the code. [20:03:08] Krinkle ^^ [20:04:12] paladox: I'm aware of that. I've read the announcement the day it came out. [20:04:22] Krinkle oh ok. [20:15:46] !log rebooting integration-slave-trusty-1016 unreachable somehow [20:15:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:17:54] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Differential-Beta, 10Mobile-App-Goals, 06Wikipedia-Android-App-Backlog: Investigate migrating the Wikipedia Android App to Differential - https://phabricator.wikimedia.org/T134505#2286639 (10Niedzielski) @mmodell, @thcipriani There do... [20:21:45] RECOVERY - SSH on integration-slave-trusty-1016 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 (protocol 2.0) [20:22:31] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Differential-Beta, 10Mobile-App-Goals, 06Wikipedia-Android-App-Backlog: Investigate migrating the Wikipedia Android App to Differential - https://phabricator.wikimedia.org/T134505#2286670 (10demon) >>! In T134505#2286639, @Niedzielski... [20:24:51] 06Release-Engineering-Team, 06Operations, 10Traffic, 10Wikimedia-General-or-Unknown: Inconsistently unable to download https://releases.wikimedia.org/mediawiki/1.26/mediawiki-1.26.2.tar.gz (returns zero-byte response) - https://phabricator.wikimedia.org/T135038#2286678 (10hashar) [20:26:11] !log rebooting integration-slave-trusty-1016 is back up [20:26:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:29:34] thcipriani: I now have to resist using `scap say` all day long :) [20:29:46] bd808: :D [20:30:31] this was, of course, the intention of the feature: work grinds to a halt. [20:31:00] "S.C.A.P.: someone can always pontificate" [20:31:13] ^ that was twentyafterfour 's [20:35:23] sounds good [20:37:33] hashar: Hi, could you have a look at https://gerrit.wikimedia.org/r/#/c/288128/ and merge please. [20:43:06] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #20: 04FAILURE in 2 min 6 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/20/ [20:48:11] 06Release-Engineering-Team, 10MediaWiki-extensions-General-or-Unknown, 05Release: Notice: Unable to unserialize: [-1]. Expected ':' but got '1'. in /srv/mediawiki/php-1.28.0-wmf.1/includes/objectcache/RedisBagOStuff.php on line 313 - https://phabricator.wikimedia.org/T134923#2282493 (10demon) Yeah unserializ... [20:50:49] I stole pontificate from epriestley, he named an event-emitter method in phabricator's javascript as 'SomeClass.pontificate()' and I lol'd [20:55:11] paladox: ah decoupling [20:55:34] paladox: might be good. I am working this week on migrating PHP jobs to Nodepool instance though [20:55:46] will try to have a look at your patch though [20:55:53] hashar: Yep, i wasen't sure if there was a cleaner way. Im also not sure about composer [20:55:55] Ok thanks [20:58:00] hashar: Looks like we have to migrate jscs soon to eslint, jscs was discontinued early this year. [20:58:07] And moved to eslint [20:59:25] (03PS2) 10Hashar: Allow decoupling of npm* and rake-jessie and jshint and jsonlint tests [integration/config] - 10https://gerrit.wikimedia.org/r/288128 (https://phabricator.wikimedia.org/T134946) (owner: 10Paladox) [21:00:24] hashar: ^^ thanks for reviewing [21:00:27] paladox: going to deploy ur change [21:00:29] looks fine [21:00:37] hashar: Thanks :) [21:01:39] ARRGGH [21:01:43] * RoanKattouw shakes fist at releng [21:02:16] (03CR) 10Hashar: [C: 032] "Deployed Jenkins jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/288128 (https://phabricator.wikimedia.org/T134946) (owner: 10Paladox) [21:02:28] James_F ^^ [21:02:30] I thought I'd avoided our cross-wiki-notifications-by-default release being delayed by the last deployment hold, and now there's another one :( [21:02:38] hashar: :) [21:02:47] RoanKattouw: well we can push code [21:02:54] RoanKattouw: and have the whole site to die :(( [21:03:19] Yeah, I know it's the responsible thing to do etc [21:03:24] there is rather concerning notice which might well explode everything :( [21:03:25] (03Merged) 10jenkins-bot: Allow decoupling of npm* and rake-jessie and jshint and jsonlint tests [integration/config] - 10https://gerrit.wikimedia.org/r/288128 (https://phabricator.wikimedia.org/T134946) (owner: 10Paladox) [21:03:38] and last week was a potentially large perf regression :/ [21:03:39] Oh https://phabricator.wikimedia.org/T134976 is bad actualyl [21:03:46] turned out to be a false alarm though [21:03:51] Oh [21:04:02] and yeah that SQL spike is concerning [21:04:59] paladox: I have pushed the Jenkins jobs and refreshed Zuul. Can you recheck oojs-ui and verify it works fine please ? ;-} [21:05:01] Yeah Chad mentioned the redis error but the SQL thing looks more alarming to me [21:05:17] hashar: Ok, i will do that now. [21:05:18] Thanks [21:05:26] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/288128 #T134946 [21:05:27] T134946: Move OOUI out of the MediaWiki gate-and-submit queue - https://phabricator.wikimedia.org/T134946 [21:05:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:05:55] paladox: I will most probably mass decouple a lot more repos. But I would like a test to cover that [21:06:04] i.e. make sure we do not couple repos again [21:06:18] hashar: Oh, ok. [21:06:22] Im testing on https://gerrit.wikimedia.org/r/#/c/288100/ [21:06:27] Thanks for merging. [21:07:24] paladox: I will most probably mass decouple a lot more repos. But I would like a test to cover that [21:07:28] i.e. make sure we do not couple repos again [21:07:31] hashar_ Ok. [21:07:47] hashar_ Im testing on https://gerrit.wikimedia.org/r/#/c/288100/ [21:07:57] hashar_ I was unsure about composer [21:08:04] RoanKattouw: I think you can see us as the folks making sure the carpet is clean both above and under (since folks tends to push dirt under the carpet) [21:08:09] Would we have to decouple composer to [21:08:17] RoanKattouw: and once it is all nice and shinny, you get to push your feature over the nice red carpet ;-} [21:08:21] Yes :) [21:08:24] * hashar_ qualifies as a Janitor [21:08:58] https://integration.wikimedia.org/ci/job/oojs-ui-npm-node-4.3/1/ [21:09:03] hashar ^^ [21:09:16] what I hope is we never end up in a position were we blindly hold deployment for frivolous reasons [21:09:54] neat! one more task aced by paladox ! [21:10:00] :) [21:10:25] Would composer need to be prefix or is composer already decouplked [21:10:29] decoupled [21:10:41] hashar ^^ [21:11:54] yeah would need it to be decouplable as needed [21:12:08] one of the issue with zuul templates is that it takes the last part of the gerrit repo name [21:12:15] eg mediawiki/core yields name = core [21:12:34] I have a patch somewhere to introduce longname = mediawiki-core [21:12:46] so in the Zuul templates we could use something like : - {longname}-composer-hhvm [21:12:55] which would trigger mediawiki-core-composer-hhvm [21:13:54] there was also a suggestion of extracting those templates job for each repos from Zuul [21:13:58] and craft a JJB project file [21:14:07] that would magically generate all the jobs we need based on the Zuul config [21:14:09] Ok [21:14:17] saving us from having to update both Zuul layout and JJB config files :D [21:14:27] in other words [21:14:34] if in zuul you have: [21:14:35] - name: mediawiki/core [21:14:37] template: [21:14:41] - name: composer [21:14:45] Oh [21:14:55] with that template having a job {longname}-composer-hhvm [21:15:04] we could have a script that generate a dummy JJB file with something like: [21:15:05] project: [21:15:21] name: mediawiki-core # that is the repo name in Zuul [21:15:22] jobs: [21:15:47] - {longname}-composer-hhvm # extracted from Zuul template 'composer' [21:15:54] which would then generates mediawiki-core-composer-hhvm [21:16:03] Ok, that would solve alot of problems. But one thing it may break some users patches if they require another patch and doint use [21:16:12] Depends-On: [21:16:12] actually all the above should be copy pasted in a task [21:16:31] Oh [21:16:50] Should i go and do composer now for oojs [21:16:54] oojs-ui [21:16:55] i mnean [21:16:57] mean [21:17:35] 10Continuous-Integration-Config, 07WorkType-NewFunctionality: Generate JJB jobs from the Zuul layout/templates definition - https://phabricator.wikimedia.org/T135059#2287036 (10hashar) [21:17:42] Also is there a way we can add more ci-trusty istances [21:17:44] Please [21:17:45] https://integration.wikimedia.org/ci/job/oojs-ui-npm-node-4.3/1/ [21:17:47] paladox: yeah you can do oojs-ui the same way [21:17:52] Ok thanks [21:17:57] I will go do that now :) [21:17:58] will deploy it tomorrow [21:18:01] Ok [21:18:05] cause Iam gonna sleep for now [21:18:09] Ok [21:18:11] for the ci-trusty images [21:18:15] there is a base pool of 2 instances [21:18:18] ci-jessie have 10 [21:18:18] Yep [21:18:24] and the overall pool is 20 instances max [21:18:29] Oh [21:18:39] so that is: 10 jessie + 2 trusty + 8 open slots [21:18:43] So we could add 8 more trusty instances [21:18:46] Oh [21:18:55] the open slots can be taken by instances being spawned , waiting for deletion [21:18:57] or running [21:19:05] the system looks at jobs waiting in Zuul [21:19:18] Oh [21:19:23] if it detects there is like 5 jobs asking for ci-trusty instances, it will spawn a few more above the 2 offered by default [21:19:41] so it would spawn a few more (not sure how many, from 3 to 5 I guess) [21:20:04] it takes a few seconds for the system to react [21:20:07] Oh [21:20:10] and an instance takes 30s - 1 minute to boot [21:20:13] :) [21:20:27] so in case there is a bunch of jobs asking for ci-trusty that might take a couple minutes before they start [21:20:37] all of that shared in the same pool with ci-jessie instances [21:20:56] anyway [21:21:06] https://phabricator.wikimedia.org/T133911 asks to get 10 of each and a max of 40 instances [21:21:09] Ok [21:21:27] I have pushed / remembered people about that task yet though [21:22:11] tgr: I love your comment on https://gerrit.wikimedia.org/r/#/c/288308/1/includes/objectcache/RedisBagOStuff.php,cm :-} [21:22:11] hashar_ Looks like your an hour ahead of me https://phabricator.wikimedia.org/T135059 im bst [21:22:22] tgr: I am definitely no more savvy in MediaWiki coding :-( [21:22:24] british summer time. [21:26:04] :) [21:26:13] ;D [21:26:21] I am crashing to bed! [21:26:25] thanks again for your patches paladox [21:26:37] Your welcome and bye [21:32:25] (03PS1) 10Paladox: Decouple composer package test for oojs-ui [integration/config] - 10https://gerrit.wikimedia.org/r/288320 (https://phabricator.wikimedia.org/T134946) [21:33:26] (03CR) 10jenkins-bot: [V: 04-1] Decouple composer package test for oojs-ui [integration/config] - 10https://gerrit.wikimedia.org/r/288320 (https://phabricator.wikimedia.org/T134946) (owner: 10Paladox) [21:41:44] (03PS2) 10Paladox: Decouple composer package test for oojs-ui [integration/config] - 10https://gerrit.wikimedia.org/r/288320 (https://phabricator.wikimedia.org/T134946) [21:42:42] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [21:42:51] (03CR) 10jenkins-bot: [V: 04-1] Decouple composer package test for oojs-ui [integration/config] - 10https://gerrit.wikimedia.org/r/288320 (https://phabricator.wikimedia.org/T134946) (owner: 10Paladox) [21:44:02] (03PS3) 10Paladox: Decouple composer package test for oojs-ui [integration/config] - 10https://gerrit.wikimedia.org/r/288320 (https://phabricator.wikimedia.org/T134946) [21:44:49] (03CR) 10jenkins-bot: [V: 04-1] Decouple composer package test for oojs-ui [integration/config] - 10https://gerrit.wikimedia.org/r/288320 (https://phabricator.wikimedia.org/T134946) (owner: 10Paladox) [21:47:07] (03PS4) 10Paladox: Decouple composer package test for oojs-ui [integration/config] - 10https://gerrit.wikimedia.org/r/288320 (https://phabricator.wikimedia.org/T134946) [21:47:59] (03CR) 10jenkins-bot: [V: 04-1] Decouple composer package test for oojs-ui [integration/config] - 10https://gerrit.wikimedia.org/r/288320 (https://phabricator.wikimedia.org/T134946) (owner: 10Paladox) [21:48:04] 10Beta-Cluster-Infrastructure: deployment-tin ssh: Connection closed by UNKNOWN - https://phabricator.wikimedia.org/T134777#2276692 (10Mattflaschen) I've gotten these intermittently too. [21:48:46] (03CR) 10Paladox: "@Hashar I'm not sure why it isen't finding the template." [integration/config] - 10https://gerrit.wikimedia.org/r/288320 (https://phabricator.wikimedia.org/T134946) (owner: 10Paladox) [22:00:01] Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #18: 04FAILURE in 8 min 0 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/18/ [22:12:06] ostriches: Any objections to me putting wmf1 of Echo in the wmf23 branch in today's SWAT? [22:12:17] (essentially circumventing the train halt for Echo) [22:12:55] I ask because today is literally the most inconvenient day all quarter for the train to stop [22:12:59] (for me) [22:13:20] That's fine by me, I'm not doing today's swat :) [22:13:35] OK [22:13:46] Just making sure you don't mind the circumvention aspect of it :) [22:16:10] (03PS1) 10Thcipriani: Update sync-wikiversions to subcommand use [tools/release] - 10https://gerrit.wikimedia.org/r/288327 [22:17:50] (03CR) 10Chad: [C: 032] "I don't use this, but lgtm :)" [tools/release] - 10https://gerrit.wikimedia.org/r/288327 (owner: 10Thcipriani) [22:18:20] thcipriani: I think it's actually kind of a redundant script now since group* dblists exist for group1 now [22:19:37] I just use updateWikiversions [22:19:40] ostriches: eh, it's got some nice things yet still [22:20:17] checks special:version, writes commit messages, git sanity checks [22:20:30] not that it's a very onerous process at this point. [22:21:39] twentyafterfour: what do you think about https://phabricator.wikimedia.org/E179#2013? IMO an "open end" is more an advantage [22:22:23] legoktm: About the feedback here https://gerrit.wikimedia.org/r/#/c/208519/38/includes/CentralAuthHooks.php [22:22:41] do i move it to the extension funtion [22:22:43] function [22:22:47] instead of callback. [22:22:52] Or what do i do. [22:23:24] (03Merged) 10jenkins-bot: Update sync-wikiversions to subcommand use [tools/release] - 10https://gerrit.wikimedia.org/r/288327 (owner: 10Thcipriani) [22:26:34] legoktm: Ive opened https://phabricator.wikimedia.org/T135075 for that. [22:28:19] 06Release-Engineering-Team, 10MediaWiki-extensions-General-or-Unknown, 13Patch-For-Review, 05Release: Notice: Unable to unserialize: [-1]. Expected ':' but got '1'. in /srv/mediawiki/php-1.28.0-wmf.1/includes/objectcache/RedisBagOStuff.php on line 313 - https://phabricator.wikimedia.org/T134923#2282493 (10A... [22:32:43] Luke081515: I'm fine with making it open-ended but some reviewers may not have that much time, so I don't think we need a formal time limit but also no formal expectation for amount of time spent either [22:32:55] e.g. neither a minimum nor maximum time limit [22:34:59] ok [22:35:11] that's the advantage if we compare it to SWAT :D [22:35:36] 10Beta-Cluster-Infrastructure: deployment-tin ssh: Connection closed by UNKNOWN - https://phabricator.wikimedia.org/T134777#2276692 (10mmodell) I think this is a race condition caused by my patch (cherry-picked on beta) I'll just abandon it. [22:49:02] (03PS15) 10Paladox: Fix dirty VisualEditor submodule [integration/config] - 10https://gerrit.wikimedia.org/r/262432 (https://phabricator.wikimedia.org/T121479) [22:49:19] (03CR) 10Paladox: "Rebased." [integration/config] - 10https://gerrit.wikimedia.org/r/262432 (https://phabricator.wikimedia.org/T121479) (owner: 10Paladox) [22:52:17] (03PS16) 10Paladox: Lets install MySQL before installing extension and extensions dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/264333 [23:21:30] (03Abandoned) 10Paladox: Add npm entry point [integration/composer] - 10https://gerrit.wikimedia.org/r/267623 (owner: 10Paladox) [23:21:35] (03Abandoned) 10Paladox: [integration/composer] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/267624 (owner: 10Paladox) [23:21:44] (03CR) 10Paladox: "recheck" [integration/composer] - 10https://gerrit.wikimedia.org/r/267623 (owner: 10Paladox) [23:29:41] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Ownership of Selenium tests - https://phabricator.wikimedia.org/T134492#2287667 (10Jdlrobson) Is it possible to send these test results to an IRC channel rather than email? [23:37:08] 10Beta-Cluster-Infrastructure: Create zero.wikimedia.beta.... instance to test zero - https://phabricator.wikimedia.org/T135082#2287717 (10Yurik) [23:38:04] 10Beta-Cluster-Infrastructure: Create zero.wikimedia.beta.... instance to test zero - https://phabricator.wikimedia.org/T135082#2287731 (10Krenair) We already have a zero.wikimedia.beta.wmflabs.org, it's just completely broken at the moment [23:43:27] 10Beta-Cluster-Infrastructure: Fix zero.wikimedia.beta.... instance to test zero - https://phabricator.wikimedia.org/T135082#2287759 (10Yurik) [23:56:01] 06Release-Engineering-Team, 10MediaWiki-extensions-General-or-Unknown, 13Patch-For-Review, 05Release: Notice: Unable to unserialize: [-1]. Expected ':' but got '1'. in /srv/mediawiki/php-1.28.0-wmf.1/includes/objectcache/RedisBagOStuff.php on line 313 - https://phabricator.wikimedia.org/T134923#2282493 (10C...