[01:01:26] 10Gerrit, 10UI-Standardization, 13Patch-For-Review: Make gerrit colors align with WikimediaUI color palette - https://phabricator.wikimedia.org/T158298#3035303 (10demon) 05Open>03declined Generally speaking, I'm not opposed to making custom CSS adjustments when we've got specific issues to fix--and if we... [02:27:58] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:44:29] paladox are you around at the moment? [03:03:00] RECOVERY - Puppet run on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [06:39:52] Project selenium-Wikibase » chrome,test,Linux,contintLabsSlave && UbuntuTrusty build #273: 04FAILURE in 1 hr 59 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/273/ [08:48:08] 10Deployment-Systems, 10Scap, 13Patch-For-Review: Update Debian Package for Scap3 - https://phabricator.wikimedia.org/T127762#3035819 (10fgiunchedi) 05Open>03Resolved [09:32:44] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: zuul-merger git-daemon process is not started properly by systemd ? - https://phabricator.wikimedia.org/T157785#3035848 (10hashar) [11:05:09] 06Release-Engineering-Team, 06Operations, 05DC-Switchover-Prep-Q3-2016-17: Understand the preparedness of misc services for datacenter switchover - https://phabricator.wikimedia.org/T156937#3036009 (10jcrespo) [11:14:54] 10Continuous-Integration-Infrastructure: operations/software repo commits always getting -1 from Jenkins-bot - https://phabricator.wikimedia.org/T158406#3036070 (10Legoktm) [11:28:45] 10Continuous-Integration-Infrastructure: operations/software repo commits always getting -1 from Jenkins-bot - https://phabricator.wikimedia.org/T158406#3036114 (10hashar) a:03hashar I am lunching but I am 99% sure it is due to T157818. Fix is to `rm -fr /srv/zuul/git/operations/software` on both contint1001... [11:29:04] 10Continuous-Integration-Infrastructure: operations/software repo commits always getting -1 from Jenkins-bot - https://phabricator.wikimedia.org/T158406#3036119 (10hashar) [11:29:06] 10Continuous-Integration-Infrastructure, 07Zuul: zuul-merger fails when repository names overlaps - https://phabricator.wikimedia.org/T157818#3017592 (10hashar) [12:04:53] 10Continuous-Integration-Infrastructure: operations/software repo commits always getting -1 from Jenkins-bot - https://phabricator.wikimedia.org/T158406#3036249 (10jcrespo) If you mean that it happens from time to time, and that is already tracked on, T157818, this can be closed as soon as the specific issue (no... [12:05:23] (03PS2) 10Zfilipin: Move Rake and RuboCop dependency from repositories to mediawiki/selenium [selenium] - 10https://gerrit.wikimedia.org/r/338137 (https://phabricator.wikimedia.org/T158326) [12:44:41] (03PS8) 10Zfilipin: Upgrade to Selenium 3 [selenium] - 10https://gerrit.wikimedia.org/r/336824 (https://phabricator.wikimedia.org/T157695) [12:49:04] 10Continuous-Integration-Infrastructure, 07Zuul: zuul-merger fails when repository names overlaps - https://phabricator.wikimedia.org/T157818#3036345 (10hashar) [12:53:19] 10Continuous-Integration-Infrastructure, 07Zuul: zuul-merger fails when repository names overlaps - https://phabricator.wikimedia.org/T157818#3036357 (10hashar) [12:53:22] 10Continuous-Integration-Infrastructure: operations/software repo commits always getting -1 from Jenkins-bot - https://phabricator.wikimedia.org/T158406#3036353 (10hashar) 05Open>03Resolved The issue was on contint2001. The operations/software/cumin got cloned first and thus operations/software could not be... [12:55:47] (03PS9) 10Zfilipin: Upgrade to Selenium 3 [selenium] - 10https://gerrit.wikimedia.org/r/336824 (https://phabricator.wikimedia.org/T157695) [13:02:41] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#2955587 (10chasemp) The fix here is actually a bit of a misnomer, and whi... [13:30:43] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool quota bump - https://phabricator.wikimedia.org/T158320#3036425 (10chasemp) > This task is to raise the pool from 19 instances to 25 > Instances 29 > 102400 kB = 25 instances * 4 GB/inst * 1024kB/GB > 118784 kB = 29 i... [13:40:46] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool quota bump - https://phabricator.wikimedia.org/T158320#3036440 (10hashar) I gave too much details I guess. The request is to bump the quota of instances to 29. That will let us have 25 instances + 2 snapshots +2 extra for pot... [13:43:50] 10Continuous-Integration-Infrastructure: operations/software repo commits always getting -1 from Jenkins-bot - https://phabricator.wikimedia.org/T158406#3036441 (10Marostegui) Thanks!! [13:53:33] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#3036456 (10hashar) >>! In T155820#3036397, @chasemp wrote: > The fix here... [14:06:33] 10Continuous-Integration-Infrastructure: operations/software repo commits always getting -1 from Jenkins-bot - https://phabricator.wikimedia.org/T158406#3036481 (10jcrespo) Also thanks, with this documentation, we will be able to fix it ourselves with confidence if it happens again, until the parent is resolved! [14:07:58] !log integration: deleting "repository" instance. No time to figure out how to ship Sonatype Nexus to it. T147635 [14:08:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:08:01] T147635: Investigate again a central cache for package managers - https://phabricator.wikimedia.org/T147635 [14:08:07] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Investigate again a central cache for package managers - https://phabricator.wikimedia.org/T147635#3036487 (10hashar) The instance worked on labs, I used with a few jenkins jobs and it works properly in the few use cases... [14:09:07] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Investigate again a central cache for package managers - https://phabricator.wikimedia.org/T147635#3036488 (10hashar) 05Open>03stalled Stalled until we get bandwidth to figure out how to ship Sonatype Nexus in a bett... [14:10:55] PROBLEM - Host repository is DOWN: CRITICAL - Host Unreachable (10.68.18.179) [14:21:00] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07Technical-Debt: Migrate CI labs slaves to use /srv instead of /mnt - https://phabricator.wikimedia.org/T146381#3036502 (10hashar) I have rebased the couple pending patches in Gerrit. Poked Giuseppe about it on https://gerrit.wikimedia.org/r/#/c/... [14:25:07] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 07Upstream, 15User-zeljkofilipin: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561#3036511 (10hashar) [14:25:10] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Packaging, 13Patch-For-Review: Create a Debian package for https://github.com/mozilla/geckodriver for at least Debian Jessie - https://phabricator.wikimedia.org/T137797#3036509 (10hashar) 05Open>03declined From https://gerrit.wi... [14:30:55] Linking dependencies: [1/14078] [14:30:57] npm is crazy [14:39:23] hashar: Can we regenerate doxygen for older branches easily? [14:39:43] Reedy: yes [14:40:02] though I can not remember off hand how the doxygen doc is generated [14:40:14] I think I made a jenkins job to just poll the mediawiki/core on an hourly basis [14:40:37] heh [14:41:22] hm no [14:41:30] it still happens after a commit is merged by triggering https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-publish/ [14:41:41] what is wrong with the docs ? [14:42:18] [forkpty: Too many open files in system] [14:42:18] bah [14:42:20] my mac is dead [14:42:23] https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-publish/ [14:42:27] (pending—Waiting for next available executor on ci-jessie-wikimedia) [14:42:39] https://doc.wikimedia.org/mediawiki-core/1.27.0/php/ApiCreateAccount_8php_source.html [14:42:39] https://doc.wikimedia.org/mediawiki-core/1.27.1/php/ApiCreateAccount_8php_source.html [14:42:56] That class shouldn't exist in 1.27.0 [14:52:45] bah [14:52:49] dirty workspace I guess [14:53:25] guess I can try regenerating the old tags [14:53:37] the job nowadays runs on Nodepool instances, so the workspace is clear or should be [15:15:45] hashar: Want me to file a TODO task for doing regeneration? [15:16:04] yup :) [15:16:12] never hurts to have a task [15:16:32] gotta trigger them via zuul [15:17:07] so this way i can use the task to document how to do it [15:17:12] which should be something like: [15:17:12] :) [15:18:21] zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/core --changes [15:18:47] eek [15:18:52] actually for tags that is a reference update [15:19:05] so same but with zuul enqueue-ref [15:19:52] 06Release-Engineering-Team, 06Labs, 06Operations: contintcloud project thinks it is using 206 fixed-ip quota errantly - https://phabricator.wikimedia.org/T158350#3036573 (10Andrew) thanks for troubleshooting -- I'll dig in the source and try to see how it's computing that quota count. [15:20:50] 10Continuous-Integration-Infrastructure: Regenerate doxygen pages for mw core - https://phabricator.wikimedia.org/T158423#3036574 (10Reedy) [15:20:58] Not sure that's the right tag rbh [15:20:59] *tbh [15:31:51] Reedy: lets try with 1.27.1 [15:35:39] paladox got a min? [15:36:27] zuul enqueue-ref --trigger gerrit --pipeline publish --project mediawiki/core --ref refs/tags/1.27.1 --newrev 5a9183baf6cd304f4c0e64b83826a9390d50ac01 [15:36:30] 10Continuous-Integration-Infrastructure: Regenerate doxygen pages for mw core - https://phabricator.wikimedia.org/T158423#3036595 (10hashar) [15:36:30] tryin [15:36:53] Reedy: running on https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-publish/17787/console [15:36:58] and the tag shows up on the zuul status page [15:37:00] hashar looks interesting ill let you know if anything on my end explodes on user-side [15:37:02] hopefully I haven't screwed it [15:37:30] hashar: PM urgent [15:37:35] 0:00:18.675 DEBUG:zuul.Cloner:Fetched ref refs/tags/1.27.1 from mediawiki/core [15:37:35] 00:00:18.675 DEBUG:zuul.Repo:Checking out 5a9183baf6cd304f4c0e64b83826a9390d50ac01 [15:38:52] 10Continuous-Integration-Infrastructure: Regenerate doxygen pages for mw core - https://phabricator.wikimedia.org/T158423#3036609 (10hashar) [15:40:19] (03PS1) 10Hashar: operations/puppet: drop erblint-HEAD/pplint-HEAD [integration/config] - 10https://gerrit.wikimedia.org/r/338369 (https://phabricator.wikimedia.org/T154894) [15:41:19] (03PS1) 10Hashar: operations/puppet: drop typo and check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/338370 [15:43:45] (03CR) 10Hashar: [C: 032] operations/puppet: drop erblint-HEAD/pplint-HEAD [integration/config] - 10https://gerrit.wikimedia.org/r/338369 (https://phabricator.wikimedia.org/T154894) (owner: 10Hashar) [15:43:54] (03CR) 10Hashar: [C: 032] operations/puppet: drop typo and check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/338370 (owner: 10Hashar) [15:44:53] (03Merged) 10jenkins-bot: operations/puppet: drop erblint-HEAD/pplint-HEAD [integration/config] - 10https://gerrit.wikimedia.org/r/338369 (https://phabricator.wikimedia.org/T154894) (owner: 10Hashar) [15:45:33] (03Merged) 10jenkins-bot: operations/puppet: drop typo and check pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/338370 (owner: 10Hashar) [15:56:40] Project beta-scap-eqiad build #142749: 04FAILURE in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/142749/ [15:57:46] 15:56:39 15:56:39 Check 'Logstash Error rate for deployment-mediawiki04.deployment-prep.eqiad.wmflabs' failed: ERROR: 75% OVER_THRESHOLD (Avg. Error rate: Before: 0.00, After: 4.00, Threshold: 1.00) [15:57:50] That can't be good [15:58:43] Reedy looks pretty normal... for a fail [15:59:55] Also, that's not 75% over threshold [15:59:58] It's 300% [16:00:13] Reedy i'd say something broke [16:07:25] Yippee, build fixed! [16:07:26] Project beta-scap-eqiad build #142750: 09FIXED in 2 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/142750/ [16:09:05] hashar question, will gerrit and/or phab automatically re encode repo into utf-8 if i change the config to utf-8 [16:12:34] Zppix: I don't know what that means [16:13:08] git handles text files as binary blobs [16:13:26] hashar the repo i have was encoded in utc-9 for some reason, and i changed it to utf-8 via repo config on phab will it automatically change the files or no? [16:13:48] It won't [16:13:55] Zppix: Recode them using notepad++ or something, and commit [16:14:07] 10Continuous-Integration-Infrastructure: Regenerate doxygen pages for mw core - https://phabricator.wikimedia.org/T158423#3036574 (10hashar) https://doc.wikimedia.org/mediawiki-core/1.27.1/php/ has been regenerated. https://doc.wikimedia.org/mediawiki-core/1.27.1/php/ApiCreateAccount_8php_source.html now yields... [16:14:14] Reedy: mw 1.27.1 doc is fixed :}}}}}}} [16:14:51] hashar: Is that the first one? :) [16:15:05] hashar: As it was 1.27.0 that was wrong... :P [16:15:35] But if your docs are right... I don't mind doing some myself [16:16:56] Reedy: sure! [16:17:32] zuul enqueue-ref lets us simulate a Gerrit events for a reference that got updated [16:17:43] typically when a tag is pushed to gerrit it creates a ref-updated events [16:18:13] has the project, the reference 'refs/tags/1.27.1', the old revision the reference which would be 00000000000000 for a creation [16:18:36] and the new revision the reference points ( 5a9183baf6cd304f4c0e64b83826a9390d50ac01 for 1.27.1 tag) [16:18:43] so what I did is head to mediawiki/core [16:18:47] refreshed the tags with git fetch --tags [16:18:59] then asked for the commit of 1.27.1 : git rev-parse refs/tags/1.27.1 [16:19:04] copy pasted it and passed it as --newrev [16:21:04] PROBLEM - Puppet run on deployment-copper is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:24:39] Reedy: some doc above ^^^ [16:27:41] Whee [16:27:48] Is it on wikitech or similar too? [16:29:27] i think all the docs could probably use a update [16:29:27] upstream doc I guess [16:30:04] Reedy: https://docs.openstack.org/infra/zuul/client.html [16:30:19] note how enqueue-ref is not in their doc bah [16:30:36] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:30:37] one more patch to add :} [16:32:58] (03PS1) 10Hashar: Switch remaining puppet modules to rake [integration/config] - 10https://gerrit.wikimedia.org/r/338378 (https://phabricator.wikimedia.org/T154894) [16:39:49] 06Release-Engineering-Team, 06Labs, 06Operations: contintcloud project thinks it is using 206 fixed-ip quota errantly - https://phabricator.wikimedia.org/T158350#3036804 (10Andrew) Usually you can force quota recalculation with MariaDB [nova]> select * from quota_usages where project_id='contintcloud'; In... [16:39:58] (03CR) 10Hashar: [C: 032] Switch remaining puppet modules to rake [integration/config] - 10https://gerrit.wikimedia.org/r/338378 (https://phabricator.wikimedia.org/T154894) (owner: 10Hashar) [16:46:11] (03Merged) 10jenkins-bot: Switch remaining puppet modules to rake [integration/config] - 10https://gerrit.wikimedia.org/r/338378 (https://phabricator.wikimedia.org/T154894) (owner: 10Hashar) [16:47:20] 6 mins to merge that really jenkins :P [16:50:12] Zppix hi [16:50:17] just saw your pings [16:50:36] i've been out all day today :) [16:52:24] paladox do you know if tools db's for enwiki can be query for usernames and edit summaries or is that only accessable through the enwiki api [16:52:33] queried* [16:52:59] Not sure as i've never used en wiki's api's [16:53:16] though i do know of them. I just find it hard to navigate the api's. [16:54:37] ugh ok [16:56:56] 06Release-Engineering-Team, 06Operations, 10Phabricator: Phabricator: Make sure phabricator works properly including our puppet roles on jessie - https://phabricator.wikimedia.org/T158434#3036857 (10Paladox) [17:01:03] RECOVERY - Puppet run on deployment-copper is OK: OK: Less than 1.00% above the threshold [0.0] [17:01:19] 06Release-Engineering-Team, 06Operations, 10Phabricator: Phabricator: Make sure phabricator works properly including our puppet roles on jessie - https://phabricator.wikimedia.org/T158434#3036884 (10Paladox) [17:05:33] RECOVERY - Puppet run on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:08:39] (03PS1) 10Hashar: [cumin] non voting debian-glue job [integration/config] - 10https://gerrit.wikimedia.org/r/338390 [17:09:37] (03CR) 10Hashar: [C: 032] [cumin] non voting debian-glue job [integration/config] - 10https://gerrit.wikimedia.org/r/338390 (owner: 10Hashar) [17:11:15] (03Merged) 10jenkins-bot: [cumin] non voting debian-glue job [integration/config] - 10https://gerrit.wikimedia.org/r/338390 (owner: 10Hashar) [17:40:12] 06Release-Engineering-Team, 06Operations, 10Phabricator: Phabricator: Make sure phabricator works properly including our puppet roles on jessie - https://phabricator.wikimedia.org/T158434#3036948 (10Paladox) p:05Triage>03High [17:42:37] 06Release-Engineering-Team, 06Operations, 10Phabricator, 10hardware-requests, 10ops-eqiad: replacement hardware for iridium (phabricator) - https://phabricator.wikimedia.org/T156970#3036950 (10Paladox) [18:07:43] 06Release-Engineering-Team, 06Labs, 06Operations: contintcloud project thinks it is using 206 fixed-ip quota errantly - https://phabricator.wikimedia.org/T158350#3036994 (10Andrew) I restarted nova-network and it looks like nova is cleaning up those leaks now. I'll keep an eye out, but I've reduced the quot... [18:58:59] PROBLEM - Puppet run on deployment-mira is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:05:40] 06Release-Engineering-Team, 06Labs, 06Operations, 13Patch-For-Review: contintcloud project thinks it is using 206 fixed-ip quota errantly - https://phabricator.wikimedia.org/T158350#3037145 (10Andrew) 05Open>03Resolved I cleaned up about 100 leaks, like this: update fixed_ips a, instances b set a.inst... [19:30:11] aezharh [19:30:14] I gotta fix stuff again :D [19:33:59] RECOVERY - Puppet run on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:11] 10Continuous-Integration-Infrastructure: Regenerate doxygen pages for mw core - https://phabricator.wikimedia.org/T158423#3037333 (10hashar) I have misread the original task. Triggering 1.27.0 now: zuul enqueue-ref --trigger gerrit --pipeline publish --project mediawiki/core --ref refs/tags/1.27.0 37fc6bf0dd1a... [20:35:28] (03Abandoned) 10Hashar: (WIP) Timed build from Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/338179 (https://phabricator.wikimedia.org/T158054) (owner: 10Hashar) [20:47:10] bd808: hello if you are still around. Do you know which puppet version is in MediaWiki vagrant jessie branch ? [20:47:59] we are installing Puppet from the wmf apt repo. let me check what version [20:48:46] * bd808 waits for his laptop to decide to actually run the command he typed [20:48:51] :D [20:50:31] turns out vagrant seems to work more or less on my box [20:50:33] arar https://atlas.hashicorp.com/debian/contrib-jessie64 :D [20:52:15] heh. apparently my jessie vm was totaly dead. its provisioning now. I think we should be installing 3.8.5 based on https://apt.wikimedia.org/wikimedia/pool/main/p/puppet/ [20:52:52] yeah that is what I would guess [20:53:04] if you happen to have jessie-wikimedia/main prioritized [20:53:04] This is a thing that I 'fixed' in the jessie branch. Instead of trying to find base images with Puppet for all the VM runtimes I added a shell script provisioning step that installs puppet from our repo [20:53:06] or something [20:53:29] also I was quickly looking at the puppet modules [20:53:39] there is a large overlap with what is in operations/puppet [20:53:53] well there is some overlap [20:53:58] but for that I dont have a good solution though maybe puppet Environments might work [20:54:21] eg use operatioons/puppet by default, and when existing the module from a mediawiki-vagrant environment [20:54:25] but that it is lot of work [20:54:49] and really not very useful honestly [20:54:58] mw-vagrant is not for testing prod puppet [20:55:27] yeah yeah [20:55:32] and working around all the problems we would have with mixing their modules and ours is more work than anyone will ever do [20:55:36] I wsa trying to find out whether some bits could be reused as-is [20:55:46] true [20:56:03] we have copied things in before [20:56:10] both directions actually [20:56:32] Ori prototyped several things in mw-vagrant that then went up to ops/puppet [20:57:00] we have a few submodules that are shared too, but I think general the prod techops folks don't like doing that [20:57:00] I see mw vagrant has the dev stage of our pipeline :] [20:57:27] with all the docker/containers talks going on, maybe eventually dev and prod will end up merging [20:57:32] and like the rest of the pipeline, it's a barely maintained volunteer project ;) [20:57:45] hehe :D [20:58:01] * bd808 will not be holding his breath for Docker to magically fix anything [20:58:20] I cant tell for Docker itself [20:58:29] I find it confusing and trying to do toooo many things [20:58:31] My personal experience with Docker has been pretty bleak [20:58:39] but surely providing small containers easilyl would be a net win [20:58:47] be it with docker or a bash script around lxc :] [20:58:57] shurg. containers are just tarballs with more crap in them [20:59:14] *shrug [20:59:22] The executable 'bsdtar' Vagrant is trying to run was not [20:59:23] bohhh [20:59:28] configuration is still the hard thing [20:59:43] that's 90% of waht mw-vagrant does [20:59:59] (and of courses vagrant download the box again bah) [21:16:28] oh I got a bug! [21:16:50] something already listening on port 8080 on the guest [21:19:23] hashar do you guys want releng added to projects on T158448 [21:19:23] T158448: Jouncebot: Crashes when issued a command. - https://phabricator.wikimedia.org/T158448 [21:21:37] Zppix thats maintained by bd808. Unless im wrong. [21:22:18] me and others. let me check the error logs. [21:23:22] bd808 fyi i forgot to mention it does it if command is pm'd or in the irc channel itself [21:23:32] even after being issued a refresh command then the now [21:23:57] its not check to make sure there is something in the list [21:24:16] pasted the error into the bug [21:24:35] bd808 ?? [21:24:59] Zppix: https://phabricator.wikimedia.org/T158448#3037391 [21:25:10] upcoming is an empty list [21:25:14] and cuasing the crash [21:25:59] bd808 and let me guess theres something broken on how it should then handle that? [21:26:10] o/ i'd like to ssh into the android CI box and see if there's an issue with sdk updates that's causing our ongoing problems with the periodic CI job. if someone has a minute, would you mind adding me as a project member to https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration? [21:26:27] (User:Mholloway) [21:26:28] Zppix it probaly was never cought as there was always deployments on there. [21:26:37] mdholloway: I should be able to... [21:26:40] mdholloway hey never seen you around how you doing? [21:26:45] Reedy: thanks! [21:27:07] paladox ah, well good thing i love breaking i mean looking at deployment windows using jouncebot [21:27:25] ok [21:27:38] Zppix: i'm good, and you? [21:28:06] mdholloway: https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Integration&diff=prev&oldid=1539588 [21:28:17] mdholloway good, breaking jouncebot like usual [21:28:17] I think that's enough... Need to wait for things to propogate [21:28:28] Let me know if you need adding to admins instead [21:28:31] /aswell [21:28:43] Reedy: awesome, thanks again! i'll give it a bit [21:30:01] bd808: so apparently puppet 3.7.2 from jessie/main at least for the first provisionning [21:30:52] hashar: oh... I guess it would vary based on the base image. If there is already puppet installed it won't be upgraded. We could change that to check for a minimum version. [21:31:30] hashar: the provision script that is run is support/puppet-bootstrap.sh [21:31:47] I am waiting for the first puppet to run [21:32:00] on the CI image I made a first puppet manifest that set all the apt thing [21:32:00] it's guard condition is "which puppet" [21:32:08] apt-get update, and then does the actual run [21:35:09] what exactly is castor? [21:35:48] CAche STORage [21:36:07] it is a lame rsync based system to cache various packages [21:36:24] a job such as the npm ones would do: [21:36:50] rsync castor:/cache/job/repo/branch /home/jenkins [21:36:56] in attempt to restore the cache [21:36:59] then npm install [21:37:00] why have a seprate instance for castor considering it runs rarely (from what i see) [21:37:26] if the job is triggered in gate and submit AND is a success, then we trigger: rsync /home/jenkins castor:/cache/job/repo/branch [21:37:30] thus saving the cache centrally [21:37:33] it is really terrible [21:37:36] * paladox is figuring how to fix polygerrit to support /r/ [21:38:13] paladox repo link and url to the thing itself? [21:38:32] hashar ah, i guess that would be true :P [21:38:39] Zppix https://gerrit-new.wmflabs.org/?polygerrit=1 [21:38:57] that has a live demo of it. I am figuring out locally how i can get it to work. [21:39:30] * paladox has a patch but it only fixes loading resources. [21:39:37] it dosent fix the api. [21:41:41] i get erros like [21:41:42] GET http://localhost:8081/config/server/version 404 (Not Found) [21:41:50] locally. [21:47:37] argh zppix is gone :( [21:58:42] sleep time *wave* [22:57:03] Reedy: there were indeed some outdated components, but looks like I'll need to be a project admin or at least have sudo on integration-slave-jessie-android to install them [22:57:17] mdholloway: Only if you want a slap for doing so [22:57:27] Project admin would be useful... [22:57:33] But surely that is managed in puppet? :) [22:58:15] ...it probably should be. i'm not 100% sure how that would work. [22:58:24] maybe i should phab it [22:58:39] Is it package updates? [22:58:46] deb/apt? [22:58:51] nope [22:59:06] goes through the android sdk manager tool [22:59:47] i'm not sure if it's puppet-able but i'm far from the puppet expert i'd like to be [22:59:55] https://github.com/wikimedia/puppet/blob/production/modules/contint/manifests/packages/androidsdk.pp [23:00:01] https://github.com/wikimedia/puppet/tree/production/modules/contint/manifests/packages [23:00:20] yeah, i saw those files with the dependencies [23:00:37] I think it may just be ad hoc installed [23:00:39] That's crappy [23:01:05] https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/ci/slave/android.pp [23:03:26] mdholloway: looks almost certainly it's been done manually... presumably from user accounts running sudo [23:05:23] ugh, wikitech has logged me out [23:07:21] Reedy: yeah, i'd assumed it had been done manually since i wasn't able to find anything relevant in the puppet configs [23:07:36] I can't actually see anything relevant in anyones bash history either [23:07:44] other than killing/restarting a few things [23:07:51] i think hashar and niedzielski have done most work in this area [23:08:29] but don't know if they've done an sdk manager update specifically [23:08:44] it probably hasn't been done since we first got the job running, honestly [23:08:49] I can't see anything like that [23:08:52] (reading) [23:09:07] Is it a new machine ish? [23:09:26] theoretically it's supposed to update dependencies on its own, but that's clearly not happening [23:09:37] heh [23:09:43] Want me to give you admin so you can poke around a bit more? [23:09:55] I don't think it's gonna cause any problems [23:09:58] niedzielski: tl;dr i'd like to update the jenkins ci machine with the latest sdk tools, build tools, etc since they're out of date and we're trying to figure out the right way to do that [23:10:02] Might be worth still creating a task to ask the questions [23:10:24] oh, integration repo [23:10:25] * Reedy looks [23:10:39] Reedy: yea, probably, i'll go ahead and open a task for discussion [23:10:50] https://github.com/wikimedia/integration-config/blob/master/jjb/mobile.yaml [23:11:16] mdholloway: i have a very sloppy command line for it here: https://gist.github.com/niedzielski/467cc7ed9ee89040be19d57863489a82 [23:11:33] heh [23:11:37] it's sloppy because it updates everything unconditionally and will probably take over an hour [23:11:47] that's what screen is for ;) [23:13:34] mdholloway: sorry, browser crashed. i'd sooner look into T147099 i think [23:13:34] T147099: Remove Android SDK manager dependency - https://phabricator.wikimedia.org/T147099 [23:14:09] mdholloway: but the periodic build server is pretty independent of our code review and other processes, so experimentation is probably fine [23:14:34] niedzielski: ah, thanks, i didn't know we had a task for that :) [23:15:41] mdholloway: i think gradle is actually at 3.x. not sure if that would help or hurt to upgrade, but if you hit problems you might consider updating [23:19:04] * Reedy attempts to break things [23:20:46] Reedy what types of things are you hoping to break. [23:20:53] all of them [23:21:28] lol, Reedy every wikimedia server? [23:23:52] 10Continuous-Integration-Config, 06Wikipedia-Android-App-Backlog, 07Technical-Debt: Figure out the right way to keep the Android SDK manager up to date - https://phabricator.wikimedia.org/T158456#3037609 (10Mholloway) [23:24:22] niedzielski: Reedy: created T158456 about this [23:24:22] T158456: Figure out the right way to keep the Android SDK manager up to date - https://phabricator.wikimedia.org/T158456 [23:25:49] i could wait on updating stuff until we discuss there. it's not exactly urgent, just that the emulator has been pretty consistently failing to start since ~november... [23:26:41] seems update is pretty dumb [23:27:10] I'll let it run and see if it helps/hinders at the end [23:28:21] 10Continuous-Integration-Config, 06Wikipedia-Android-App-Backlog, 07Technical-Debt: Figure out the right way to keep the Android SDK up to date in CI - https://phabricator.wikimedia.org/T158456#3037632 (10Mholloway)