[00:16:18] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" components deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2485968 (10greg) //As this task is short without all of the context, I am going to paste here my answer to someone who already asked the most im... [00:21:57] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" components deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2485975 (10greg) Scare quoting "components" as well just to call it out that defining what is and isn't included here is also a part of it. My... [00:22:21] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2485976 (10greg) [00:28:22] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2485985 (10greg) Comments for the record :) The pro-activeness of Reading team to create their page is awesome and I greatly appreciate it an... [00:30:32] ostriches: fixed with https://gerrit.wikimedia.org/r/300459 follow-up. gerrit restarted and the file mode changes are applied now [00:31:22] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:31:52] Whoops [00:32:00] Thanks mutante [00:32:48] :) [00:34:18] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2485989 (10greg) [00:53:25] PROBLEM - Free space - all mounts on deployment-jobrunner01 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner01.diskspace.root.byte_percentfree (<40.00%) [00:59:49] ostriches, hi gerrit old has supported downloading in gerrit for long time, just not as good as gerrit new does it now [01:00:08] You can download straight from https://gerrit.wikimedia.org/r/#/c/300323/1/modules/gerrit/templates/error.html.erb [01:00:14] when the download button is at the top [01:00:36] mutante ^^ [01:00:57] yes, that's what i meant and used before [01:01:15] in the diff itself [01:01:29] if the file is small it doesnt become a zip and gives you raw file in browser [01:01:33] if it's bigger it zips it [01:01:51] well or multiple files if they are changed [01:01:55] oh [01:03:29] Oh herp derp, guess that should be on still. [01:03:55] * ostriches sighs [01:04:02] Will look tomorrow, I'm done for the day :p [01:04:34] ok thanks [01:05:34] yea, let's continue tomoroww. i'm also done [01:05:43] but the big one is merged :) [01:11:21] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [04:15:20] 07Browser-Tests, 10Continuous-Integration-Config, 10MediaWiki-extensions-RelatedArticles, 06Reading-Web-Backlog: RelatedArticles browser tests should run on a commit basis - https://phabricator.wikimedia.org/T120715#2486137 (10bmansurov) Given that we may remove ReadMore from wikis, I say we don't invest a... [04:18:23] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #81: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/81/ [04:24:38] 07Browser-Tests, 10Continuous-Integration-Config, 10MediaWiki-extensions-RelatedArticles, 06Reading-Web-Backlog: RelatedArticles browser tests should run on a commit basis - https://phabricator.wikimedia.org/T120715#1859792 (10Legoktm) >>! In T120715#2486137, @bmansurov wrote: > Given that we may remove Re... [06:04:39] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk - https://phabricator.wikimedia.org/T71976#2486182 (10Joe) Just FTR, this is solved and the title of the bug is misleading. Resolving. [06:05:12] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#2486184 (10Joe) [06:05:29] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk - https://phabricator.wikimedia.org/T71976#2486183 (10Joe) 05Open>03Invalid [07:18:35] PROBLEM - SSH on integration-slave-jessie-1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:16] 07Browser-Tests, 10Continuous-Integration-Config, 10MediaWiki-extensions-RelatedArticles, 06Reading-Web-Backlog: RelatedArticles browser tests should run on a commit basis - https://phabricator.wikimedia.org/T120715#2486274 (10bmansurov) >>! In T120715#2486141, @Legoktm wrote: > Err, context please? Is the... [08:27:10] 10Deployment-Systems, 10scap, 10Analytics-Cluster, 06Analytics-Kanban, and 2 others: Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2486313 (10elukey) I can see other keys expired with gpg --list-keys. @mark, @yuvipanda, @chasemp: would you mind to double check your gpg k... [08:34:26] 06Release-Engineering-Team, 15User-greg: Identify RelEng projects 'worthy' of a tech lead - https://phabricator.wikimedia.org/T139540#2435395 (10hashar) **agree** at least we have identified that the bulk of our projects are team shared and the few epic ones already have natural leaders :-} [08:49:29] 07Browser-Tests, 10Continuous-Integration-Config, 10MediaWiki-extensions-RelatedArticles, 06Reading-Web-Backlog: RelatedArticles browser tests should run on a commit basis - https://phabricator.wikimedia.org/T120715#2486342 (10Jhernandez) @bmansurov AFAIK that's about desktop only actually. It seems like i... [08:58:23] RECOVERY - SSH on integration-slave-jessie-1002 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [09:38:25] PROBLEM - Free space - all mounts on deployment-jobrunner01 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner01.diskspace.root.byte_percentfree (<22.22%) [09:50:31] jenkins seems to be down: https://integration.wikimedia.org/ci/job/mwext-testextension-php55/16972/console [09:50:36] https://gerrit.wikimedia.org/r/#/c/299743/ [09:50:52] hashar: ^ [09:54:53] Amir1: ohh [09:54:58] mysql is dead on that slave apparently [09:55:14] yeah, it can't connect to it [09:55:56] !log integration-slave-trusty-1001 service mysql start [09:56:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:59:11] 160722 6:40:24 InnoDB: 5.5.50 started; log sequence number 46210678922 [09:59:11] ERROR: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE user ADD column Show_view_priv enum('N','Y') CHARACTER SET utf8 NOT ' at line 1 [09:59:12] 160722 6:40:24 [ERROR] Aborting [09:59:12] 160722 6:40:24 InnoDB: Starting shutdown... [09:59:14] 160722 6:40:25 InnoDB: Shutdown completed; log sequence number 46210678922 [09:59:14] 160722 6:40:25 [Note] /usr/sbin/mysqld: Shutdown complete [10:00:06] Do we delete the database then allow the test to regenerate the tables? [10:00:08] hashar ^^ [10:03:45] paladox: yeah a db is created for the job [10:03:53] Oh [10:04:06] 10Continuous-Integration-Infrastructure: mysql shutdown at 6:40 on integration-slave-trusty-1001 - https://phabricator.wikimedia.org/T141083#2486432 (10hashar) [10:04:13] Amir1: looks like mysql is down everywhere bah :( [10:04:23] 10Continuous-Integration-Infrastructure: mysql shutdown at 6:40 on integration-slave-trusty-1001 - https://phabricator.wikimedia.org/T141083#2486443 (10hashar) [10:04:27] hashar oh, is it down on nodepool [10:04:42] :/ [10:05:38] 10Continuous-Integration-Infrastructure: mysql shutdown at 6:40 on integration-slave-trusty-1001 - https://phabricator.wikimedia.org/T141083#2486452 (10Ladsgroup) p:05Triage>03Unbreak! [10:06:00] 10Continuous-Integration-Infrastructure: mysql shutdown at 6:40 on integration-slave-trusty-1001 - https://phabricator.wikimedia.org/T141083#2486454 (10hashar) What ever upgraded happened on Trusty caused all mysql to die with: ``` 160722 6:48:38 InnoDB: 5.5.50 started; log sequence number 22953995199 ERROR: 10... [10:06:16] 10Continuous-Integration-Infrastructure: mysql shutdown at 6:40 on integration-slave-trusty-1001 - https://phabricator.wikimedia.org/T141083#2486456 (10Ladsgroup) All jenkins jobs involving mysql fail: https://gerrit.wikimedia.org/r/#/c/299743/ [10:06:20] !log T141083 salt -v '*slave-trusty*' cmd.run 'service mysql start' [10:06:21] T141083: mysql shutdown at 6:40 on integration-slave-trusty-1001 - https://phabricator.wikimedia.org/T141083 [10:06:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:06:40] Amir1: should be good now. Thank you for the ping! [10:07:04] you downgraded it? [10:07:07] 10Continuous-Integration-Infrastructure: mysql shutdown at 6:40 on integration-slave-trusty-1001 - https://phabricator.wikimedia.org/T141083#2486459 (10hashar) 05Open>03Resolved a:03hashar [10:07:33] hashar: thanks :) [10:10:01] !log rebooting integration-slave-jessie-1002 and integration-slave-trusty-1018 . Hang somehow [10:10:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:10:53] hashar, microsoft edge and internet explorer block debian's bug page. [10:10:58] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=708176 [10:11:40] paladox: ah neat [10:11:46] Yep [10:11:53] But i got the big red screen [10:11:53] paladox: can you add that to the task https://phabricator.wikimedia.org/T141083 ? [10:11:57] saying doint enter [10:11:58] Ok [10:12:44] 10Continuous-Integration-Infrastructure: mysql shutdown at 6:40 on integration-slave-trusty-1001 - https://phabricator.wikimedia.org/T141083#2486493 (10Paladox) Bug report at debian https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=708176 [10:12:51] hashar done ^^ [10:13:12] paladox: neat and I have poked WMF Database administrator, but then production uses MariaDB / customized package [10:13:18] so prod should be safe :D [10:13:21] Oh [10:13:23] :) [10:17:18] hashar oh yes, the old change screen in gerrit is gone in gerrit 2.12, i forgot about that. :) [10:17:41] !log Jenkins can't ssh / add slaves integration-slave-jessie-1002 or integration-slave-trusty-1018 . Apparently due to some Jenkins deadlock in the ssh slave plugin :-/ Lame way to solve it: restart Jenkins [10:17:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:18:17] paladox: production is safe ™ [10:18:31] hashar oh [10:18:37] I am still wondering how you manage to find all those useful upstream bug reports [10:18:46] Searched google [10:18:53] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [10:18:57] and was in the first page reults [10:19:00] hashar ^^ [10:19:10] y [10:19:20] :) [10:19:34] hashar What do you mean by production is safe? [10:19:38] hashar ^^ [10:19:45] PROBLEM - Puppet staleness on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [43200.0] [10:19:52] paladox: that production is not going to be impacted by the debian bug you found [10:19:55] we used mariadb [10:19:58] Oh [10:19:59] we use [10:20:00] ah [10:20:01] :) [10:20:09] and moreover, that is a custom package just for us [10:20:10] but isent mariadb based on mysql [10:20:16] and it is never magically upgraded :d [10:20:20] oh [10:20:59] hashar i guess the new change screen will cause confussion [10:21:09] people we adapt [10:21:22] but it is a really good update, just wished that it would have been made more user frendly. [10:21:30] pretty sure chad announced about it [10:21:37] And plus diffs have been tougt to be faster [10:21:59] I can load integration/config layout.yaml changes [10:22:04] finally on internet explorer [10:22:06] instantly [10:22:09] no matter what we do, there is always a few people complaining. But overall I am pretty sure the vast majority will be fine with Gerrit 2.12 [10:22:31] Also they fixed the bug in ios that made it take along time to load a diff on the iphone it is was a big file [10:22:36] Yep [10:22:58] Everyone will be interested in the web editor. [10:23:04] hashar ^^ [10:23:43] !log Jenkins has some random deadlock. Will probably reboot it [10:23:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:23:51] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:43] RECOVERY - Puppet staleness on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [3600.0] [10:28:09] zeljkof: I have restarted Jenkins in case you played with browsertests they need to be rescheduled [10:30:18] hashar: thanks for letting me know, I think nothing major is happening [10:30:36] Tobi_WMDE_SW_NA: in case one of your jobs is still running ^ [10:32:52] !log Jenkins restarted and it pooled both integration-slave-jessie-1002 and integration-slave-trusty-1018 [10:32:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:37:01] hashar :) [10:38:53] hashar but the good thing is it wont stall our mail any more [10:38:56] in gerrit 2.12 [10:41:36] hashar im wondering what you think is to be a good timeout number, since the default is 0, which means it is indefinate. [10:41:42] https://gerrit.googlesource.com/gerrit/+/a7e343131777750d11c12d06354e52aaae9badc5%5E%21/#F4 [10:47:32] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream, 07WorkType-NewFunctionality: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#2486538 (10hashar) [10:47:59] hashar: it just failed again [10:48:11] https://gerrit.wikimedia.org/r/299743 [10:49:00] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs cleanup and optional task - https://phabricator.wikimedia.org/T140235#2486545 (10Tobi_WMDE_SW) [10:49:02] 10Browser-Tests-Infrastructure, 05MW-1.27-release-notes, 13Patch-For-Review, 15User-zeljkofilipin: Remove LoginPage from mediawiki_selenium Ruby gem - https://phabricator.wikimedia.org/T127042#2486546 (10Tobi_WMDE_SW) [10:49:03] hashar https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm/18638/console [10:49:09] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-zeljkofilipin, 03Wikidata-Sprint-2016-07-19: Merge tests/browser/environments.yml and tests/browser/config/config.yml in WikidataBrowserTests - https://phabricator.wikimedia.org/T128097#2486542 (10Tobi_WMDE_SW) 05Open>03Resolved a... [10:49:50] 10Browser-Tests-Infrastructure, 10Wikidata, 13Patch-For-Review, 15User-zeljkofilipin, 03Wikidata-Sprint-2016-07-19: selenium-Wikibase Jenkins job fails with `no such file to load -- features/support/pages (LoadError)` - https://phabricator.wikimedia.org/T140096#2486549 (10Tobi_WMDE_SW) 05Open>03Resolv... [10:58:09] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream, 07WorkType-NewFunctionality: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#1387714 (10hashar) >>! In T103351#1387872, @hashar wrote: > > > Gotta report that to #upstream #jenkin... [11:23:38] 03Scap3, 06Operations, 10Ops-Access-Requests, 06Services: Allow Pchelolo to deploy services via Scap3 - https://phabricator.wikimedia.org/T141086#2486581 (10mobrovac) [11:24:06] 03Scap3, 06Operations, 10Ops-Access-Requests, 06Services: Allow Pchelolo to deploy services via Scap3 - https://phabricator.wikimedia.org/T141086#2486593 (10mobrovac) @GWicke please approve. [11:24:28] 03Scap3, 06Operations, 10Ops-Access-Requests, 06Services: Allow Pchelolo to deploy services via Scap3 - https://phabricator.wikimedia.org/T141086#2486594 (10elukey) p:05Triage>03Normal [11:37:31] 03Scap3, 06Operations, 10Ops-Access-Requests, 06Services, 13Patch-For-Review: Allow Pchelolo to deploy services via Scap3 - https://phabricator.wikimedia.org/T141086#2486636 (10mobrovac) [11:42:20] hashar also thanks for doing zuul :) :) [11:48:27] (03PS5) 10Lethexie: Single Line comments no multiple '*'. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/295895 [11:48:52] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Drop some Trusty permanent slaves from integration labs project - https://phabricator.wikimedia.org/T139535#2486678 (10hashar) 05Open>03Resolved 2 big ones got dropped. That is good enough for now. More will be deleted as jobs are sh... [12:07:04] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#2486742 (10hashar) [12:25:50] hashar: I'm seeing a lot of issues similar to https://phabricator.wikimedia.org/T109704 today.. e.g. https://integration.wikimedia.org/ci/job/mwext-testextension-php55-composer/3651/console is there a known problem currently? [12:27:51] Tobi_WMDE_SW: yeah mysql died on all instances due to an upgrade [12:28:14] Tobi_WMDE_SW: Amir1 poked about it this morning. It is solved now [12:29:20] It just happened now: https://integration.wikimedia.org/ci/job/mwext-mw-selenium-composer/4012/console [12:29:27] https://gerrit.wikimedia.org/r/#/c/299284/ [12:29:30] hashar: ^ [12:30:59] oh [12:31:08] Amir1: bah that is just that one slave :) [12:32:29] :D [12:36:06] hashar, i will be able to help test the new zuul update when ever you pull the latest from upstream :) [12:47:00] (03PS1) 10Hashar: Merge upstream bc58ea34125f11eb353abc3e5b96ac1efad06141 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300531 [12:47:02] (03PS1) 10Hashar: 2.1.0-391-gbc58ea3-wmf1precise1 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300532 [13:02:33] !log zuul rebased patch queue on tip of upstream branch and force pushed branch. c3d2810...4ddad4e HEAD -> patch-queue/debian/precise-wikimedia (forced update) [13:02:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:03:00] hashar :) [13:03:29] (03CR) 10Hashar: [C: 032 V: 032] Merge upstream bc58ea34125f11eb353abc3e5b96ac1efad06141 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300531 (owner: 10Hashar) [13:05:46] hashar did you include your patch you submited upstream? [13:18:53] (03PS1) 10Hashar: WMF: soften paramiko requirement [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300538 [13:18:55] (03PS1) 10Hashar: WMF: soften WebOb requirement [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300539 [13:18:57] (03PS1) 10Hashar: WMF: drop requirement ordereddict [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300540 [13:19:09] (03CR) 10Hashar: [C: 032 V: 032] WMF: soften paramiko requirement [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300538 (owner: 10Hashar) [13:19:13] (03CR) 10Hashar: [C: 032 V: 032] WMF: soften WebOb requirement [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300539 (owner: 10Hashar) [13:19:19] (03CR) 10Hashar: [C: 032 V: 032] WMF: drop requirement ordereddict [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300540 (owner: 10Hashar) [13:21:03] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 07Upstream, 15User-zeljkofilipin: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561#2371835 (10zeljkofilipin) a:05zeljkofilipin>03None [13:21:08] (03PS2) 10Hashar: 2.1.0-391-gbc58ea3-wmf1precise1 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300532 [13:21:24] * hashar whistles [13:21:34] paladox: which patch ? [13:21:57] hashar the one which allows us to change the default [13:21:59] gerrit time [13:22:14] ah not yet [13:22:19] :)} [13:22:23] https://review.openstack.org/#/c/343562/ [13:22:26] hashar ^^ [13:22:26] might as well cherry pick it right now before I forget [13:22:28] :) [13:22:32] Oh ha [13:24:25] hashar if you want to we can also cherry pick https://review.openstack.org/#/c/295237/ since it was +2 but not yet merged [13:24:56] But may make tests faster since they wont be waiting until the one in the front to merge. [13:24:59] (03PS1) 10Hashar: Gerrit trailing delay is now configurable [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300542 [13:25:26] (03PS3) 10Hashar: 2.1.0-391-gbc58ea3-wmf1precise1 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300532 [13:25:28] paladox: done :} [13:25:36] hashar thanks :) [13:25:38] :) [13:26:17] paladox: that would need a change in zuul.conf [13:26:20] in [gerrit] [13:26:23] event_delay = 5 [13:26:32] Oh [13:26:39] Do we submit that as a puppet patch [13:26:40] ? [13:26:43] hashar ^^ [13:27:42] * hashar tries to figure out what is going with APScheduler [13:28:52] hashar oh is that APScheduler broken again? [13:28:57] why not bump to 3.2.0 [13:30:49] OMG my wifi extender decided to disconnect my pc [13:30:51] but not my phone [13:30:53] ha [13:31:07] hashar ^^ [13:31:34] hashar are you getting any errors in APScheduler [13:31:37] ? [13:31:51] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 07Upstream, 15User-zeljkofilipin: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561#2486958 (10zeljkofilipin) Thanks @Peter, I have talked with @hashar and the current plan is to support: - Firefox 46 (current... [13:35:44] (03PS1) 10Hashar: WMF: soften pbr requirement [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300543 [13:35:46] (03PS1) 10Hashar: WMF: constraint apscheduler to <3.1.0 [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300544 [13:36:06] 07Browser-Tests, 10Browser-Tests-Infrastructure, 10MobileFrontend, 06Reading-Web-Backlog, 15User-zeljkofilipin: `Generic special page features.Search from Watchlist` test failing with Net::ReadTimeout - https://phabricator.wikimedia.org/T130971#2152450 (10zeljkofilipin) a:05zeljkofilipin>03None [13:36:32] (03CR) 10Paladox: "I thought 3.2.0 would fix it." [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300544 (owner: 10Hashar) [13:36:45] (03PS4) 10Hashar: 2.1.0-391-gbc58ea3-wmf1precise1 [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300532 [13:37:05] paladox: we currently have apscheduler 3.0.x [13:37:05] (03CR) 10Paladox: "Why not use pbr 0.8.2" [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300543 (owner: 10Hashar) [13:37:10] paladox: so I am sticking to it [13:37:11] hashar oh [13:37:18] but woulden 3.2.0 fix our problem [13:37:25] with 3.1.0 [13:37:32] and apparently 3.3.0 has been released [13:37:34] hashar ^^ [13:37:47] and stop pinging me constantly please :D [13:37:49] I am in the channel! [13:37:57] Oh sorry. [13:40:50] sorry :). [13:47:40] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #88: 04FAILURE in 3 min 39 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/88/ [14:05:49] 07Browser-Tests, 10Continuous-Integration-Config, 10MediaWiki-extensions-RelatedArticles, 06Reading-Web-Backlog: RelatedArticles browser tests should run on a commit basis - https://phabricator.wikimedia.org/T120715#2487091 (10zeljkofilipin) >>! In T120715#2483192, @Jhernandez wrote: > @bmansurov @zeljkofi... [14:13:10] 07Browser-Tests, 10MobileFrontend, 06Reading-Web-Backlog, 03Reading-Web-Sprint-77-Segmentation-fault, and 4 others: Spike [2hrs] Wikidata description browser tests do not run anywhere - https://phabricator.wikimedia.org/T137756#2487112 (10zeljkofilipin) [14:14:36] 07Browser-Tests, 10MobileFrontend, 06Reading-Web-Backlog, 03Reading-Web-Sprint-77-Segmentation-fault, and 4 others: Spike [2hrs] Wikidata description browser tests do not run anywhere - https://phabricator.wikimedia.org/T137756#2377830 (10zeljkofilipin) I am happy to pair with somebody on this as soon as I... [14:31:41] 10Continuous-Integration-Infrastructure, 06Operations, 10Zuul, 07Blocked-on-Operations: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2487152 (10hashar) @elukey proposed to review the package and we had a quick discussion about it. Turns out upgradin... [14:37:08] I mananged to git merge debian/precise-wikimedia into debian/jessie-wikim [14:37:14] debian/jessie-wikimedia [14:37:15] :) [14:37:29] hehe [14:37:53] I just need to find out how i can now pull in those debian [14:37:56] patches [14:38:07] I am not there yet :D [14:38:07] you have for debian/precise-wikimedia [14:38:10] Oh [14:38:17] Are you getting errors? [14:38:26] the patches are in debian/patches directory of the branch debian/precise-wikimedia [14:38:31] Yep [14:38:44] so when you merge the branch debian/precise-wikimedia into debian/jessie-wikimedia, that inject all the patches from debian/patches [14:38:49] and hence [14:38:49] if you rebuild the package [14:38:54] Oh [14:38:56] it should have all the proper patches (hopefully) [14:39:03] Oh [14:39:06] How do i rebuild [14:39:08] please [14:39:08] ? [14:39:47] https://www.mediawiki.org/wiki/Continuous_integration/Zuul#new_package ? :D [14:39:53] Oh thanks [14:39:57] and want to make sure your local "upstream" branch points to the proper commit [14:40:03] eg the one that is in debian/changelog [14:40:12] But dosent that build it [14:40:17] I am more or less off, doing some python right now [14:40:27] How do i regenerate the patches in debian/patches [14:40:30] Ok [14:49:57] Yay i got the debian/patches in from cherry picking your commit [14:50:47] Im building the dpkg now [14:51:18] (03CR) 10EBernhardson: [C: 032] Single Line comments no multiple '*'. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/295895 (owner: 10Lethexie) [14:52:09] (03Merged) 10jenkins-bot: Single Line comments no multiple '*'. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/295895 (owner: 10Lethexie) [14:53:20] Its built now [14:56:55] Ive applied it no [14:56:56] now [14:58:19] It works http://gerrit-jenkins.wmflabs.org/job/composer-gerrit-test/68/console [15:07:47] :-} [15:08:22] :) [15:08:27] It works [15:08:35] including changing the time it detects gerrit [15:08:42] I set it to 3 secs and so is fast now [15:09:04] I see you only changed it for zuul, but doint zuul-merger and zuul-server have to be set [15:10:19] (03CR) 10Hashar: "check experimental" [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300532 (owner: 10Hashar) [15:11:18] I doint think there are any experimental jobs for ^^ [15:21:08] hashar oh wait zuul has stopped picking up changes now [15:21:13] https://gerrit-zuul.wmflabs.org/ [15:24:06] Works again [15:26:27] 03Scap3, 06Operations, 10Ops-Access-Requests, 06Services, 13Patch-For-Review: Allow Pchelolo to deploy services via Scap3 - https://phabricator.wikimedia.org/T141086#2487367 (10GWicke) Approved. [15:31:01] (03CR) 10Paladox: [C: 031] "It works." [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300532 (owner: 10Hashar) [15:37:15] (03PS6) 10Zfilipin: WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) [15:38:04] (03CR) 10jenkins-bot: [V: 04-1] WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) (owner: 10Zfilipin) [15:39:43] Yippee, build fixed! [15:39:43] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #87: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/87/ [15:47:03] (03PS1) 10Hashar: debian-glue: pass PBUILDER_USENETWORK for Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/300564 [15:48:01] (03CR) 10Hashar: [C: 032] debian-glue: pass PBUILDER_USENETWORK for Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/300564 (owner: 10Hashar) [15:48:23] paladox: thank you for the package testing :} [15:48:43] (03Merged) 10jenkins-bot: debian-glue: pass PBUILDER_USENETWORK for Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/300564 (owner: 10Hashar) [15:48:44] hashar your welcome :) [15:48:56] At least we know that it will work with gerit 2.12 [15:49:06] or at least looks like it will work [15:49:08] yeah that is great [15:49:09] :) [15:49:33] (03CR) 10Hashar: "check experimental" [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/300532 (owner: 10Hashar) [15:49:47] I am trying to sprint rush a patch that would let us build the zuul package from Gerrit / Jenkins [15:49:49] :D [15:49:58] hashar OH [15:49:59] yay [15:50:01] will probably polish that next week instead [15:50:05] Ok [15:50:05] but the idea is one would push a patch [15:50:11] Yep [15:50:12] and Jenkins will be build the package / report [15:50:16] Oh :) [15:50:29] twentyafterfour had that setup for a few debian repositories on Differential [15:50:32] and it works well [15:50:34] Oh :) [15:50:40] Will you be able to download [15:50:41] the deb [15:50:45] when it builds [15:50:47] ? [15:50:48] the trick is http://jenkins-debian-glue.org/ [15:50:55] Oh [15:50:56] and on a labs instance apply the package_builder class [15:51:01] Oh [15:51:04] that puppet class setup the images to build packages into [15:51:08] there might be some doc on wikitech [15:51:17] will update that in the zuul packaging tutorial I have to write down [15:51:30] Oh [15:51:39] :) [15:52:07] Thankyou for updating zuul today, i havent tested it on precise since i doint have a precise machine to test, but i tested on jessie and works [15:52:08] :) [15:52:25] it may even fix the problem we had the other [15:52:28] day with stale refs [15:52:37] and it not finding the HEAD [15:54:04] bah I need debian glue 14.0+ :( [15:54:48] Oh [15:57:05] 10Continuous-Integration-Infrastructure: Upgrade jenkins-debian-glue on Jessie slaves from 0.13.0 to latest (0.17.0) - https://phabricator.wikimedia.org/T141114#2487529 (10hashar) [15:57:33] 10Continuous-Integration-Infrastructure: Upgrade jenkins-debian-glue on Jessie slaves from 0.13.0 to latest (0.17.0) - https://phabricator.wikimedia.org/T141114#2487544 (10hashar) Source package https://packages.debian.org/source/sid/jenkins-debian-glue [15:59:29] (03PS1) 10Hashar: (DO NOT MERGE) fetch proper repo/heads [integration/config] - 10https://gerrit.wikimedia.org/r/300567 (https://phabricator.wikimedia.org/T117869) [15:59:55] ok I am off now [16:22:02] 06Release-Engineering-Team, 15User-greg: Identify RelEng projects 'worthy' of a tech lead - https://phabricator.wikimedia.org/T139540#2487651 (10greg) >>! In T139540#2485589, @greg wrote: > I *think* what might actually be useful is an explicit "(tech) lead" column on our quarterly goal page, eg https://www.me... [16:22:33] 06Release-Engineering-Team, 07Documentation, 15User-greg: Document tech leads for RelEng projects - https://phabricator.wikimedia.org/T139539#2487656 (10greg) Probably done by just doing eg: https://www.mediawiki.org/w/index.php?title=Wikimedia_Release_Engineering_Team%2FGoals%2F201617Q1&type=revision&diff=2... [16:37:59] !log bumping scap to v.3.2.1 on deployment-tin to test canary deploys [16:38:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:45:53] Project beta-scap-eqiad build #112303: 04FAILURE in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/112303/ [16:46:29] hmm, weird, quilt patches must not have gotten applied correctly for packaging :( [16:46:44] !log rolling back scap version to v.3.2.0 [16:46:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:53:42] !log bumping scap to v.3.2.1 on deployment-tin to test canary deploys, again [16:53:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:54:53] 07Browser-Tests, 10Continuous-Integration-Config, 10MediaWiki-extensions-RelatedArticles, 06Reading-Web-Backlog: RelatedArticles browser tests should run on a commit basis - https://phabricator.wikimedia.org/T120715#2487870 (10Jdlrobson) Given that misleading information above it's worth pointing out that... [16:55:50] Yippee, build fixed! [16:55:50] Project beta-scap-eqiad build #112304: 09FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/112304/ [17:06:05] Project beta-scap-eqiad build #112305: 04FAILURE in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/112305/ [17:13:32] 17:06:04 17:06:04 scap failed: AttributeError 'module' object has no attribute 'run_canary_checks' (duration: 01m 31s) [17:15:53] Yippee, build fixed! [17:15:53] Project beta-scap-eqiad build #112306: 09FIXED in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/112306/ [17:27:29] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2487971 (10greg) See also: https://www.mediawiki.org/wiki/Talk:Writing_an_extension_for_deployment#Support and https://www.mediawiki.org/wiki/... [17:30:57] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2487985 (10greg) And https://www.mediawiki.org/wiki/Talk:Developers/Maintainers#Kill_extensions.27_.22Maintainers.22_data points out that for... [17:31:43] ostriches, hi can we set sendemail.connectTimeout in gerrit see https://gerrit.googlesource.com/gerrit/+/a7e343131777750d11c12d06354e52aaae9badc5%5E%21/#F4 and https://phabricator.wikimedia.org/T131189 please. [17:32:20] 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2487988 (10greg) //Since this task was an outcome of the https://wikitech.wikimedia.org/w... [17:33:34] paladox: Yeah, I was just figuring out a timeout. [17:33:43] Oh, thanks :) [17:34:06] We'll probably just set it to a high value. We don't actually *need* the timeout, we just want there to be *some* so they eventually flush out if a particular e-mail gets stuck trying to connect. [17:34:10] (To avoid the bug we hit before) [17:34:17] Yep [17:34:18] :) [17:35:05] ostriches also about the downloads, it seems that we have had it enabled on the old gerrit [17:35:21] just it was hidden more and didnt allow you download as much as gerrit new is [17:35:55] mutante proposes we should disccuss disabling downloads after the gerrit migration. [17:36:54] 06Release-Engineering-Team, 06Operations, 15User-greg: Institute a weekly review of all UBN! tasks - https://phabricator.wikimedia.org/T141130#2488009 (10greg) [17:37:14] 06Release-Engineering-Team, 06Operations, 15User-greg: Institute a weekly review of all UBN! tasks - https://phabricator.wikimedia.org/T141130#2488026 (10greg) [17:37:18] 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2456573 (10greg) [17:38:14] ostriches ^^ [17:46:09] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2488072 (10greg) [17:50:20] it can be discussed of course, i just mean it can be unrelated to switch on Sunday [17:50:29] Yep [17:50:30] :) [17:50:54] that was more referring to the topics in gerrit btw [17:51:04] since we separate them in "pre" and "post" switch [17:51:53] Oh [17:56:05] Project beta-scap-eqiad build #112310: 04FAILURE in 1 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/112310/ [17:56:17] ^ me again [18:06:09] Project beta-scap-eqiad build #112311: 04STILL FAILING in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/112311/ [18:16:01] Project beta-scap-eqiad build #112312: 04STILL FAILING in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/112312/ [18:16:30] ^ failing in new ways, so there's something [18:19:44] Ok Carl [18:25:59] Yippee, build fixed! [18:25:59] Project beta-scap-eqiad build #112313: 09FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/112313/ [18:39:49] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2488481 (10greg) [18:41:50] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2488498 (10greg) [18:43:37] 06Release-Engineering-Team, 15User-greg: Identify "responsible parties" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2485829 (10greg) Removing that subtask as it is already part of a chain of work that overlaps with this but I don't want to disturb things alr... [19:06:39] hot diggity dog! 19:06:01 Executing check 'Logstash Error rate for deployment-mediawiki01.deployment-prep.eqiad.wmflabs' [19:07:05] !log beta-cluster has successfully used a canary for mediawiki deployments [19:07:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:12:47] yay! [19:40:12] 10Continuous-Integration-Infrastructure: Upgrade jenkins-debian-glue on Jessie slaves from 0.13.0 to latest (0.17.0) - https://phabricator.wikimedia.org/T141114#2488638 (10hashar) I have updated our copy in `operations/debs/jenkins-debian-glue.git` and pushed tags. Rebuild using a cowbuilder env set up via our... [19:41:35] i see you can now update jenkins-debian-glue [19:43:07] to 0.17.0 [19:43:08] :) [19:44:22]