[00:21:27] (03CR) 10Krinkle: Make ZeroBanner and MobileApps dependencies of MobileFrontend (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/307794 (https://phabricator.wikimedia.org/T144412) (owner: 10Jdlrobson) [00:44:51] (03CR) 10Jdlrobson: Make ZeroBanner and MobileApps dependencies of MobileFrontend (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/307794 (https://phabricator.wikimedia.org/T144412) (owner: 10Jdlrobson) [01:05:42] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:06:52] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:20:20] AndyRussG: You around? I'd like to have a chat with you about CentralNotice's wmf_deployment branch. Not urgent, whenever's convenient for you [01:38:26] RoanKattouw: yeah I'm here! Thx!!! Mmm maybe grab a slot on my calendar? [01:38:49] Sure [01:38:55] Yeah I just realized it's pretty late in EDT by now [01:39:05] And that I should probably eat something [01:40:28] I'll send an email instead, and if we need to talk about it in real time, hit me up on IRC tomorrow [01:40:43] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:41:14] RoanKattouw: sure! Yeah I am still working today, should try to finish something, or at least make a bit of progress... Brain power is also kinda low and distractions abound..... [01:41:30] thx much [01:41:53] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:24:25] Project selenium-QuickSurveys » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #140: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/140/ [03:04:41] (03PS1) 10Krinkle: Add jobs for labs/tools/guc (composer-php55) [integration/config] - 10https://gerrit.wikimedia.org/r/307909 [03:07:56] (03CR) 10Krinkle: [C: 032] Add jobs for labs/tools/guc (composer-php55) [integration/config] - 10https://gerrit.wikimedia.org/r/307909 (owner: 10Krinkle) [03:09:10] (03Merged) 10jenkins-bot: Add jobs for labs/tools/guc (composer-php55) [integration/config] - 10https://gerrit.wikimedia.org/r/307909 (owner: 10Krinkle) [03:09:28] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/307909 [03:09:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:36:44] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:55:13] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [04:11:43] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:30:14] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [04:30:21] 10Deployment-Systems, 03Scap3 (Scap3-MediaWiki-MVP), 10scap, 10MediaWiki-API, and 4 others: Create a script to run test requests for the MediaWiki service - https://phabricator.wikimedia.org/T136839#2600695 (10Legoktm) I wrote a rather simple autogenerated API spec based on Anomie's advice above. It requir... [06:07:51] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:34:06] Yippee, build fixed! [06:34:06] Project mediawiki-core-code-coverage build #2235: 09FIXED in 3 hr 34 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2235/ [06:50:13] Yippee, build fixed! [06:50:13] Project selenium-Wikibase » firefox,test,Linux,contintLabsSlave && UbuntuTrusty build #94: 09FIXED in 2 hr 10 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/94/ [07:12:53] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:32:12] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 07Upstream, 15User-zeljkofilipin: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561#2600815 (10Peter) FYI: Geckodriver 0.10 and the coming 1.0 release only works with Selenium 3.0 (only in beta right now), so t... [08:03:53] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:43:51] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:52:40] 10Continuous-Integration-Config, 10MediaWiki-extensions-ZeroBanner, 06Wikipedia-Android-App-Backlog, 10Wikipedia-App-MobileApp-extension, and 3 others: MobileFrontend should run the tests of ZeroBanner and MobileApps - https://phabricator.wikimedia.org/T144412#2600929 (10phuedx) [08:53:41] 10Continuous-Integration-Config, 10MediaWiki-extensions-ZeroBanner, 06Wikipedia-Android-App-Backlog, 10Wikipedia-App-MobileApp-extension, and 3 others: MobileFrontend should run the tests of ZeroBanner and MobileApps - https://phabricator.wikimedia.org/T144412#2599002 (10phuedx) @Jdlrobson: Can you guessti... [08:55:34] (03PS42) 10Zfilipin: Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) [09:04:53] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:07:04] PROBLEM - Puppet run on deployment-elastic07 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:37:45] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:42:01] RECOVERY - Puppet run on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [09:44:54] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:46:04] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:12:42] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:21:05] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [10:47:46] 10releng-201516-q4, 10Malu (Malu-Prototype): Meta: Prototype new JS framework (malu) - https://phabricator.wikimedia.org/T131121#2601135 (10zeljkofilipin) [10:57:02] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2601144 (10zeljkofilipin) a:03zeljkofilipin [11:16:42] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:25:59] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:42:28] PROBLEM - Puppet staleness on deployment-salt02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [11:51:43] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:05] 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2601226 (10Paladox) [11:57:44] 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2512615 (10Paladox) or @chasemp please ^^ I guess it may be a problem in the images. [12:05:58] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:06:21] (03CR) 10Tobias Gritschacher: "I think this can be abandoned since we made the paths relative to the env.rb file in https://gerrit.wikimedia.org/r/#/c/300508/15/tests/br" [integration/config] - 10https://gerrit.wikimedia.org/r/291907 (https://phabricator.wikimedia.org/T140096) (owner: 10Zfilipin) [12:15:57] 10Browser-Tests-Infrastructure, 10Wikidata, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2601243 (10Tobi_WMDE_SW) [12:17:36] 10Browser-Tests-Infrastructure, 10Wikidata, 15User-Tobi_WMDE_SW: Remove https://integration.wikimedia.org/ci/job/selenium-Wikidata/ - https://phabricator.wikimedia.org/T144487#2601255 (10Tobi_WMDE_SW) [12:17:48] 10Browser-Tests-Infrastructure, 10Wikidata, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2601243 (10Tobi_WMDE_SW) p:05Triage>03Normal [12:20:30] PROBLEM - Puppet run on mira is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:22:56] !log migrating deployment-tin keyholder to use base::service_unit for moritm https://gerrit.wikimedia.org/r/#/c/307510/ + reboot + keyholder arm [12:23:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:24:10] Project beta-code-update-eqiad build #119563: 15ABORTED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/119563/ [12:26:15] 10Gerrit: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2601269 (10Paladox) gerrit-test3 uses kernel Linux gerrit-test3 4.4.0-1-amd64 #1 SMP Debian 4.4.2-3+wmf3 (2016-07-28) x86_64 Debian GNU/Linux 8.5 (jessie) and gerrit-MySQL uses Linux gerrit-mysq... [12:26:30] Project beta-code-update-eqiad build #119564: 15ABORTED in 8.7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/119564/ [12:27:40] (03Abandoned) 10Zfilipin: WIP Fix selenium-Wikibase Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/291907 (https://phabricator.wikimedia.org/T140096) (owner: 10Zfilipin) [12:30:30] RECOVERY - Puppet run on mira is OK: OK: Less than 1.00% above the threshold [0.0] [12:43:40] (03PS1) 10Tobias Gritschacher: Remove job selenium-Wikidata [integration/config] - 10https://gerrit.wikimedia.org/r/307940 (https://phabricator.wikimedia.org/T144487) [12:56:15] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:06:30] 06Release-Engineering-Team (Deployment-Blockers), 13Patch-For-Review, 05Release: MW-1.28.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T142117#2601310 (10hashar) [13:31:15] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [13:35:57] (03CR) 10Bmansurov: [C: 031] Make ZeroBanner and MobileApps dependencies of MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/307794 (https://phabricator.wikimedia.org/T144412) (owner: 10Jdlrobson) [13:50:36] 10Gerrit, 06Labs: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2601400 (10Paladox) Adding labs project since were stuck until it is fixed. [13:56:43] 06Release-Engineering-Team (Deployment-Blockers), 13Patch-For-Review, 05Release: MW-1.28.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T142117#2601415 (10hashar) [13:56:59] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:57:10] 06Release-Engineering-Team (Deployment-Blockers), 13Patch-For-Review, 05Release: MW-1.28.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T142117#2523354 (10hashar) T144484 has been hot fixed, it is no more a blocker. [14:19:22] 10Gerrit: In Gerrit, clear watchlist of several accounts that are no more used (to avoid mail server errors) - https://phabricator.wikimedia.org/T113926#1680032 (10Paladox) I believe in gerrit 2.13 they have switched to saving users watches in a git repo instead of the db. which may make what your asking a littl... [14:23:18] 10Gerrit: Short commit hashes with all numbers cannot be searched using gerrit - https://phabricator.wikimedia.org/T86035#960251 (10Paladox) This looks like it is supported now. Doing https://gerrit.wikimedia.org/r/#/q/159326 resolved correctly to https://gerrit.wikimedia.org/r/#/c/159326/ can we close this ta... [14:31:56] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:54:36] 10Gerrit, 06Labs: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2601541 (10Paladox) [15:37:29] RECOVERY - Puppet staleness on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [3600.0] [15:37:50] 06Release-Engineering-Team, 06Labs, 06Operations, 10wikitech.wikimedia.org, 07LDAP: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2601696 (10Sophivorus) Thanks! So @chasemp or @Andrew told you it's not ok to change a user's UID in LDAP? [15:43:34] 10Gerrit: Short commit hashes with all numbers cannot be searched using gerrit - https://phabricator.wikimedia.org/T86035#2601741 (10Umherirrender) >>! In T86035#2601447, @Paladox wrote: > This looks like it is supported now. > > Doing https://gerrit.wikimedia.org/r/#/q/159326 resolved correctly to https://gerr... [15:43:37] 10Gerrit, 06Labs: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2601743 (10chasemp) {T144494} should be fine now [15:47:39] 10Gerrit, 06Labs: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2601760 (10Paladox) 05stalled>03Open [15:56:27] Project language-screenshots-VisualEditor » chrome,Windows 10,ci-jessie-wikimedia build #50: 15ABORTED in 7 hr 0 min: https://integration.wikimedia.org/ci/job/language-screenshots-VisualEditor/BROWSER=chrome,PLATFORM=Windows%2010,label=ci-jessie-wikimedia/50/ [15:58:51] zeljkof: 7 hours time out :( [15:59:22] Build timed out (after 420 minutes). Marking the build as aborted. [15:59:39] you can see ruby flushing the output on termination [16:02:15] hashar: argh, 6:30 was enough last time :( [16:02:21] will bump it to 8 hours o.O [16:02:48] maybe splitting the job in two (parent/child) could help out [16:02:58] one job creates screenshots, the other uploads them [16:03:16] it would still take the same amount of time, but each job would be half of the time :| [16:05:03] so sad... to get that far and then BAM! [16:05:48] zeljkof: will it reupload the snapshots or does it manage to skip them ? [16:06:22] hashar: not sure what you are asking [16:06:32] I mean [16:06:34] if you rerun it [16:06:36] will the next job reupload the existing photes? [16:06:38] photos [16:06:43] it is going to generate snapshots for everything and upload a bunch of dupes ? [16:06:48] no, it will recreate everything :| [16:12:19] zeljkof: I am pretty sure the images have the sha1 computed and that is exposed by the mw api [16:12:24] food for later :] [16:14:02] (03PS43) 10Zfilipin: Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) [16:17:41] zeljkof: https://commons.wikimedia.org/w/api.php?action=query&titles=File:VisualEditor_Media_Insert_Menu-nb.png&prop=imageinfo&iilimit=1&iiprop=timestamp|user|sha1|url :D [16:17:56] get you the sha1 and the name of user for the last version [16:18:05] copy paste for later reference :] [16:18:15] hashar: thanks :) [16:18:15] you should be able to pass a list of titles [16:18:29] to retriever the sha1 of a batch of images in a single api query [16:18:42] and have the user field included to confirm the last ver is from the screenshot bot [16:18:43] not sure if the sha will be the same, since we take the screenshot and crop it every time, a few pixels could be different [16:18:54] maybe the mediawiki_api gem will make that very easy [16:18:56] but good idea [16:19:18] well do the sha1 compute after the crop / just before the upload [16:19:27] I am off [16:19:37] tonight I will look at rspec shared_context ( https://relishapp.com/rspec/rspec-core/v/3-4/docs/example-groups/shared-context ) :D [16:22:45] 10Continuous-Integration-Config, 10MediaWiki-extensions-ZeroBanner, 06Wikipedia-Android-App-Backlog, 10Wikipedia-App-MobileApp-extension, and 3 others: MobileFrontend should run the tests of ZeroBanner and MobileApps - https://phabricator.wikimedia.org/T144412#2601879 (10Jdlrobson) p:05High>03Normal gu... [16:23:00] (03PS4) 10Jdlrobson: Make ZeroBanner and MobileApps dependencies of MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/307794 (https://phabricator.wikimedia.org/T144412) [16:23:24] hasharAway: [16:23:40] ^ any chance you could help me merge the above. I'm short on people who understand that part of the stack. [16:23:46] (if you come back today :)) [16:26:34] 10Beta-Cluster-Infrastructure, 06Operations, 05Prometheus-metrics-monitoring: deploy prometheus node_exporter and server to deployment-prep - https://phabricator.wikimedia.org/T144502#2601885 (10fgiunchedi) [16:36:57] (03CR) 10Legoktm: [C: 032] Make ZeroBanner and MobileApps dependencies of MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/307794 (https://phabricator.wikimedia.org/T144412) (owner: 10Jdlrobson) [16:37:56] (03Merged) 10jenkins-bot: Make ZeroBanner and MobileApps dependencies of MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/307794 (https://phabricator.wikimedia.org/T144412) (owner: 10Jdlrobson) [16:38:14] <3 legoktm [16:38:18] !log deploying https://gerrit.wikimedia.org/r/307794 [16:38:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:39:00] jdlrobson: np [17:07:25] 10Continuous-Integration-Config, 10MediaWiki-extensions-ZeroBanner, 06Reading-Web-Backlog, 06Wikipedia-Android-App-Backlog, and 4 others: MobileFrontend should run the tests of ZeroBanner and MobileApps - https://phabricator.wikimedia.org/T144412#2602032 (10MBinder_WMF) [17:08:01] PROBLEM - Puppet run on deployment-elastic07 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:10:53] PROBLEM - Puppet run on deployment-elastic05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:12:43] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:15:29] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:15:32] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:15:36] PROBLEM - Puppet run on deployment-parsoid09 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:15:36] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:15:50] PROBLEM - Puppet run on deployment-kafka04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:15:54] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:16:28] PROBLEM - Puppet run on mira is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:16:38] PROBLEM - Puppet run on deployment-elastic08 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:17:06] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:17:08] 10Gerrit: Short commit hashes with all numbers cannot be searched using gerrit - https://phabricator.wikimedia.org/T86035#2602120 (10Paladox) Oh I see now, doing a test on https://gerrit-review.googlesource.com/#/c/84672/ which is running gerrit 2.13 seems to work. So looks like when we update to gerrit 2.13 wi... [17:20:08] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:20:10] PROBLEM - Puppet run on deployment-elastic06 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:20:18] PROBLEM - Puppet run on deployment-mediawiki03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:20:34] PROBLEM - Puppet run on deployment-apertium01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:20:40] PROBLEM - Puppet run on deployment-poolcounter02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:20:45] hrm [17:20:47] PROBLEM - Puppet run on deployment-sca03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:20:49] PROBLEM - Puppet run on deployment-zookeeper01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:20:49] PROBLEM - Puppet run on deployment-fluorine02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:20:49] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:20:57] PROBLEM - Puppet run on deployment-pdfrender is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:21:05] PROBLEM - Puppet run on deployment-zotero01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:21:35] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:21:35] > Error 400 on SERVER: Failed to execute generator /usr/local/sbin/make-labs-root-password [17:22:15] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:22:59] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:25:09] PROBLEM - Puppet run on deployment-pdf02 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [17:25:17] ugh [17:25:21] PROBLEM - Puppet run on deployment-stream is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:25:23] PROBLEM - Puppet run on deployment-conftool is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:25:30] thcipriani it was https://gerrit.wikimedia.org/r/307086 [17:25:36] greg-g ^^ [17:25:37] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:25:44] PROBLEM - Puppet run on deployment-mx is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:25:48] PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:27:14] why is our puppetmaster testing positive for labs_puppet_master? [17:27:36] greg-g https://gerrit.wikimedia.org/r/307980 [17:28:31] heh, so, that's a bit out of order eh? :) [17:28:35] will the next run fix it? [17:29:06] Not sure [17:29:23] that was merged at 18:09pm my time [17:29:31] which was 20mins ago [17:30:08] let's see (is my guess) [17:30:08] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:30:08] PROBLEM - Puppet run on deployment-db2 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:30:10] PROBLEM - Puppet run on deployment-ircd is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:30:12] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:30:20] OK [17:30:22] PROBLEM - Puppet run on deployment-kafka03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:30:28] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:30:50] PROBLEM - Puppet run on deployment-imagescaler01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:30:56] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:31:02] PROBLEM - Puppet run on deployment-kafka01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:31:02] PROBLEM - Puppet run on deployment-sentry01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:31:04] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:31:19] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:31:37] hrm, just tried to re-run on one of the failing machines, no joy. That var is in /etc/puppet/hieradata as well, FWIW [17:32:21] I guess those two patches broke it then. Need reverting or fixed? [17:32:27] thcipriani: blugh [17:32:49] andrewbogott: ^ fyi, we're getting Error 400 on SERVER: Failed to execute generator /usr/local/sbin/make-labs-root-password [17:33:43] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [0.0] [17:34:16] thcipriani: where are you seeing that? [17:34:53] andrewbogott: the error? sudo tail /var/log/puppet.log ? [17:34:59] like, what host? [17:35:13] oh, heh, pick one: deployment-mediawiki03 [17:35:19] PROBLEM - Puppet run on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:35:23] PROBLEM - Puppet run on deployment-ms-be02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:35:25] all deployment-prep hosts looks like [17:35:29] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:35:45] PROBLEM - Puppet run on deployment-cache-upload04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:35:53] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:35:55] PROBLEM - Puppet run on deployment-memc05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:35:55] PROBLEM - Puppet run on deployment-db03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:35:59] PROBLEM - Puppet run on deployment-puppetmaster is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:36:05] do y'all have labs_puppet_master set in the custom hiera for that project? [17:36:19] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:36:57] Not that I'm aware of... looks like that's coming back labs-puppetmaster-eqiad.wikimedia.org using hiera command line stuff, but I could be invoking something weird. [17:37:15] ah, crap, we do [17:37:19] https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep [17:37:35] thcipriani: ok, well, don't do that :) [17:37:49] heh, removing [17:38:39] maybe I need to change that to use a setting like "real_labs_puppet_master_for_all_labs_regardless_of_what_you_set_for_your_project_or_this_instance_seriously_dont_override_this" [17:39:02] explicit is better than implicit [17:39:06] yuvipanda: I think it's resolved… deployment_prep had labs_puppet_master overridden in local hiera for no particular reason [17:39:13] whatever it is I didn't do it [17:39:25] ah [17:39:30] ok [17:39:31] cool :) [17:39:44] yuvipanda: what do you think I should name that setting so it's unambigiously NOT something you set in project- or instance-local hiera? [17:40:28] PROBLEM - Puppet run on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:40:29] andrewbogott labs_default_puppetmaster? [17:40:32] or labs::puppetmaster::default? [17:40:34] PROBLEM - Puppet run on deployment-db04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:40:45] hrm. do I need to reset something in /etc/puppet now? Still seeing the failure. [17:41:06] labs_default_puppetmaster seems to invite overriding just as much... [17:41:29] also: what's the magic for looking up hiera stuff? I'm using: sudo RUBYLIB=/var/lib/puppet/lib hiera labs_puppet_master ::hostname=deployment-puppetmaster.deployment-prep.eqiad.wmflabs --config /etc/puppet/hiera.yaml --debug [17:41:30] thcipriani: maybe need to restart your puppetmaster? I don't know how to refresh your local hiera settings [17:41:58] thcipriani: I don't remember… ori wrote a tool that was something like just 'hieralookup' I think [17:42:46] thcipriani: what do you like better… labs_global_puppet_master or labs_default_puppetmaster? [17:42:48] or something else? [17:43:46] labs_global_puppetmaster seems less ambiguous. Maybe something less ambiguous than "Labs"...don't know what though :) [17:44:16] well I got a good run on deployment-puppetmaster [17:45:08] PROBLEM - Puppet run on deployment-mediawiki01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:45:11] and deployment-mediawiki03 should start seeing RECOVERYs soon [17:45:27] andrewbogott: thanks for your help [17:46:26] thcipriani: https://gerrit.wikimedia.org/r/#/c/307994/1 <- ? [17:47:33] +1'd [17:50:10] ok, merged — looks harmless to me, hopefully it doesn't break your stuff again [17:52:05] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [17:52:35] 06Release-Engineering-Team, 15User-greg: Create P&T offsite slides (due 9/12) - https://phabricator.wikimedia.org/T144511#2602204 (10greg) [17:54:31] 10Gerrit: Short commit hashes with all numbers cannot be searched using gerrit - https://phabricator.wikimedia.org/T86035#2602220 (10Umherirrender) >>! In T86035#2602120, @Paladox wrote: > Oh I see now, doing a test on https://gerrit-review.googlesource.com/#/c/84672/ which is running gerrit 2.13 seems to work.... [17:55:07] RECOVERY - Puppet run on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:19] RECOVERY - Puppet run on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:31] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:36] RECOVERY - Puppet run on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:36] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:40] RECOVERY - Puppet run on deployment-poolcounter02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:50] RECOVERY - Puppet run on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:50] RECOVERY - Puppet run on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:50] RECOVERY - Puppet run on deployment-zookeeper01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:54] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:56] RECOVERY - Puppet run on deployment-pdfrender is OK: OK: Less than 1.00% above the threshold [0.0] [17:56:00] RECOVERY - Puppet run on deployment-puppetmaster is OK: OK: Less than 1.00% above the threshold [0.0] [17:56:04] RECOVERY - Puppet run on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:56:30] RECOVERY - Puppet run on mira is OK: OK: Less than 1.00% above the threshold [0.0] [18:00:08] RECOVERY - Puppet run on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:00:10] 10Continuous-Integration-Config, 10MediaWiki-extensions-ZeroBanner, 06Reading-Web-Backlog, 06Wikipedia-Android-App-Backlog, and 4 others: MobileFrontend should run the tests of ZeroBanner and MobileApps - https://phabricator.wikimedia.org/T144412#2602228 (10Jdlrobson) 05Open>03Resolved THis is working:... [18:00:33] RECOVERY - Puppet run on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:00:37] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [18:00:45] RECOVERY - Puppet run on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [18:00:47] RECOVERY - Puppet run on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:00:49] RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [18:01:35] RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:02:15] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [18:02:59] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:06] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:08] RECOVERY - Puppet run on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:10] RECOVERY - Puppet run on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:12] RECOVERY - Puppet run on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:20] RECOVERY - Puppet run on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:24] RECOVERY - Puppet run on deployment-conftool is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:30] RECOVERY - Puppet run on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:47] RECOVERY - Puppet run on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:57] RECOVERY - Puppet run on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:06:03] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:06:19] RECOVERY - Puppet run on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:08:17] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:10:11] RECOVERY - Puppet run on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [18:10:21] RECOVERY - Puppet run on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:10:25] RECOVERY - Puppet run on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:10:29] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:10:49] RECOVERY - Puppet run on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:10:54] RECOVERY - Puppet run on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:11:02] RECOVERY - Puppet run on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:11:02] RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:11:20] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:13:06] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:13:44] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:15:22] RECOVERY - Puppet run on deployment-ms-be02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:15:28] RECOVERY - Puppet run on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:15:34] RECOVERY - Puppet run on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:15:44] RECOVERY - Puppet run on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:15:56] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:15:56] RECOVERY - Puppet run on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [18:17:15] 10Gerrit, 06Labs: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2602247 (10Dzahn) @chasemp @paladox confirmed, working now (yea, Access denied = "works" in this case, before it was all timeout) thank you! ``` dzahn@gerrit-test3:~$ mysql -h gerrit-mys... [18:19:29] hrmm, how do you get the cherry-pick command for latest gerrit? [18:20:29] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:22:41] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [18:32:03] !log T130861: Cherry picking https://gerrit.wikimedia.org/r/#/c/282466/ to deployment-puppetmaster [18:32:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:34:17] !log T130861: Restarting Cassandra on deployment-restbase0[1-2] [18:34:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:43:43] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:53:04] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [18:56:39] 10Beta-Cluster-Infrastructure, 06Labs: puppet::self hosts now have two servers set - https://phabricator.wikimedia.org/T144108#2602426 (10Andrew) 05Open>03declined I guess I don't care about this for now. [18:57:16] 10Beta-Cluster-Infrastructure, 06Labs: puppet::self hosts now have two servers set - https://phabricator.wikimedia.org/T144108#2602442 (10yuvipanda) They shouldn't anymore, I fixed 'em all on tuesday. [18:58:26] !log T130861: De-cherry-picking https://gerrit.wikimedia.org/r/#/c/282466/ [18:58:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:59:15] (03CR) 10Krinkle: [C: 031] Enable PSR2.Methods.FunctionClosingBrace sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/307046 (owner: 10Legoktm) [19:00:41] !log T130861: Restarting Cassandra on deployment-restbase0[1-2] [19:00:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:01:23] (03CR) 10Polybuildr: [C: 032] Enable PSR2.Methods.FunctionClosingBrace sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/307046 (owner: 10Legoktm) [19:02:08] (03Merged) 10jenkins-bot: Enable PSR2.Methods.FunctionClosingBrace sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/307046 (owner: 10Legoktm) [19:16:47] 10Gerrit: Short commit hashes with all numbers cannot be searched using gerrit - https://phabricator.wikimedia.org/T86035#2602580 (10Paladox) Yep but it should work? [19:23:41] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:14] Yay it created the db now [19:25:35] Almost time to call https://phabricator.wikimedia.org/T141803 a sucess :) [19:25:42] ostriches do i run reindex manually? [19:26:04] Notice: /Stage[main]/Gerrit::Jetty/Exec[reindex_gerrit_jetty]/returns: [2016-09-01 19:24:22,095] [main] INFO com.google.gerrit.server.project.ProjectCacheWarmer : Loading project cache [19:26:04] Notice: /Stage[main]/Gerrit::Jetty/Exec[reindex_gerrit_jetty]/returns: fatal: fetch failure on changes [20:06:00] RoanKattouw hi, im wondering do you remeber the patch you said didnt correct show properly in the new gerrit 2.12 update? [20:06:02] it was echo [20:06:08] and cut of part of it at the end [20:06:17] i have another test website up running gerrit 2.13 [20:06:38] ostriches also we wont be able to use polygerrit with that cookie thing unless we build it with there open patch [20:06:48] http://gerrit-new.wmflabs.org/ [20:18:12] 03Scap3, 06Services, 10service-runner, 10service-template-node, 15User-mobrovac: Enable config deploys for service::node services - https://phabricator.wikimedia.org/T144542#2602980 (10mobrovac) [20:18:19] 03Scap3, 06Services, 10service-runner, 10service-template-node, 15User-mobrovac: Enable config deploys for service::node services - https://phabricator.wikimedia.org/T144542#2602993 (10mobrovac) p:05Triage>03High [20:33:24] paladox: Hmm no I don't remember which patch it was offhand [20:33:36] Oh but I think I linked to it in a bug report in the Gerrit bug tracker somewhere [20:33:45] You may be able to find it by searching for bugs I reported / commented on? [20:34:03] Ok thanks [20:37:41] (03CR) 10Hoo man: "I guess "selenium-Wikibase" supersedes this?" [integration/config] - 10https://gerrit.wikimedia.org/r/307940 (https://phabricator.wikimedia.org/T144487) (owner: 10Tobias Gritschacher) [20:40:30] greg-g sucess it worked with a remote db now, i just need to add a few changes to a paste and then we can say sucess with gerrit being able to install on labs. [20:44:51] Strange im getting this error [20:44:52] java.lang.IllegalArgumentException: Invalid plugin file /var/lib/gerrit2/review_site/plugins/reviewnotes.jar: cannot get plugin name [20:47:03] Caused by: java.io.FileNotFoundException: /var/lib/gerrit2/review_site/plugins/reviewnotes.jar (Permission denied) [20:48:31] Yippee, build fixed! [20:48:31] Project performance-webpagetest-wpt-org build #1854: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/performance-webpagetest-wpt-org/1854/ [20:59:02] Yay it worked [20:59:14] https://phabricator.wikimedia.org/P3637 are now hopefully all whats needed [20:59:20] including https://phabricator.wikimedia.org/P3939 [21:01:27] ostriches it works ^^ :) [21:01:39] https://gerrit.git.wmflabs.org/ [21:03:08] Yippee, build fixed! [21:03:08] Project selenium-Wikidata » firefox,test,Linux,contintLabsSlave && UbuntuTrusty build #104: 09FIXED in 2 hr 13 min: https://integration.wikimedia.org/ci/job/selenium-Wikidata/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/104/ [21:04:42] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:16:05] PROBLEM - Puppet run on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:16:16] man, why you break again puppet? [21:16:22] * greg-g moves on to other things [21:22:47] greg-g: that is just a one off error [21:22:54] at least on ms-fe01 [21:23:42] and deployment-conf03 is broken [21:24:56] :/ [21:25:57] 06Release-Engineering-Team, 15User-greg: "publish" draft Q2 goals (due 9/2) - https://phabricator.wikimedia.org/T144549#2603259 (10greg) [21:34:42] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:39:05] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 13Patch-For-Review, 07Regression: Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault" - https://phabricator.wikimedia.org/T142158#2603294 (10Krinkle) Could be unrelated but just noticed another one of these in a simila... [22:10:58] 10Gerrit: Fine-tune gerrit cache settings - https://phabricator.wikimedia.org/T141065#2603412 (10demon) 05Open>03Resolved This is better now. Can always revisit later. For posterity, here's our current cache stats: ``` Name |Entries | AvgGet |Hit Ratio|... [22:16:32] 10Gerrit, 10GitHub-Mirrors, 10RESTBase: Gerrit and github out of sync for RESTbase repository - https://phabricator.wikimedia.org/T115298#1720562 (10demon) I'm not understanding the bug here. Gerrit does not (and will not) sync from Github, it goes the other direction. If you develop on Github, you'll need t... [22:20:51] 10Gerrit, 10GitHub-Mirrors, 10RESTBase: Gerrit and github out of sync for RESTbase repository - https://phabricator.wikimedia.org/T115298#2603498 (10demon) p:05Triage>03Normal [22:26:42] 10Gerrit, 10GitHub-Mirrors, 10RESTBase: Gerrit and github out of sync for RESTbase repository - https://phabricator.wikimedia.org/T115298#2603535 (10Pchelolo) 05Open>03Invalid Certainly 'invalid', we push to gerrit manually when it's time for deploy. [22:33:28] 06Release-Engineering-Team, 03releng-201617-q1, 15User-greg: Perform a technical debt analysis of software and services maintained by WMF Release Engineering - https://phabricator.wikimedia.org/T138225#2603557 (10greg) [22:33:30] 06Release-Engineering-Team, 03releng-201617-q1, 15User-greg: Make tech debt analysis working documents public/documented - https://phabricator.wikimedia.org/T144059#2603555 (10greg) 05Open>03Resolved https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Project/201617Q1-TechDebt [22:33:54] greg-g ok success now, i have shown mutante all the command i have used now [22:33:58] and it is a success [22:34:15] gerrit now offically works on labs with these guides [22:34:15] https://phabricator.wikimedia.org/P3637 [22:34:20] https://phabricator.wikimedia.org/P3939 [22:34:21] 06Release-Engineering-Team, 03releng-201617-q1, 15User-greg: Perform a technical debt analysis of software and services maintained by WMF Release Engineering - https://phabricator.wikimedia.org/T138225#2393799 (10greg) 05Open>03Resolved a:03greg Documented, goals/plans changed for next quarter. Calling... [22:34:23] ostriches ^^ [22:35:09] Ahahahahaha! I figured out how to push all refs/changes/* to github :p [22:35:15] * ostriches outsmarted github [22:35:26] RECOVERY - Host deployment-parsoid05 is UP: PING OK - Packet loss = 0%, RTA = 0.92 ms [22:35:33] for i in {00..99}; do git push https://demon@github.com/wikimedia/mediawiki refs/changes/$i/*:refs/changes/$i/*; done [22:35:34] I win! [22:35:41] * ostriches pours himself a drink [22:35:58] Yay [22:36:19] Didn't even need to be all that clever once I thought about it [22:36:27] LOL [22:36:30] I'm on 08 [22:36:36] ostriches are you doing mw core? [22:36:37] Just gotta iterate all the way to 99 [22:36:39] Yep [22:36:41] Yay [22:36:42] :) [22:36:44] I'll do puppet next [22:36:46] yayyay [22:37:06] Im creating the how to setup gerrit guide on wikitech now [22:37:09] with our paste [22:38:02] I saw. The init/reindex should still be puppet managed, but whatever I'll live. [22:38:31] Yep [22:40:46] 10Gerrit, 06Labs: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2603572 (10Dzahn) We went through the instructions one more time and edited them slightly. It's done now, we have instructions how to get puppet up and running with just puppet that are r... [22:41:07] PROBLEM - Host deployment-parsoid05 is DOWN: CRITICAL - Host Unreachable (10.68.16.120) [22:47:21] ostriches greg-g https://wikitech.wikimedia.org/wiki/How_to_setup_Gerrit_in_Labs [22:47:22] :) [22:48:29] 10Gerrit, 06Labs: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2603582 (10Paladox) This is all fixed now yay. Thanks @Dzahn and @demon Link to the guide on the wiki is https://wikitech.wikimedia.org/wiki/How_to_setup_Gerrit_in_Labs [22:48:52] 10Gerrit, 06Labs: fix puppet issues when applying role::gerrit::server in labs - https://phabricator.wikimedia.org/T141803#2603583 (10Paladox) 05Open>03Resolved a:03Paladox Closing as resolved now. [22:49:08] Yay resolved now ^^ [22:56:09] RoanKattouw hi, looks like gerrit 2.13 resolves that issue any ways with it cutting half of the content [22:56:20] and so when we upgrade we wont need that workaround : [22:56:22] :) [23:00:28] Nice [23:00:38] Cause the workaround breaks contextual commenting, sadly [23:00:55] Which actually makes me wonder if we shouldn't undo it for that reason [23:01:40] RoanKattouw if you want we can?, or is there a better css rule we can use? [23:01:49] I don't think so [23:01:58] Ok [23:02:48] If you want RoanKattouw we can revert that css change? but it hopefully should be fixed in gerrit 2.13, i carnt be 100% sure but, testing with http://gerrit-new.wmflabs.org/r/#/c/1/2..3/index.php works [23:06:51] Yeah it was weird, it didn't always happen, lemme find my thing [23:07:01] Ok [23:09:03] RoanKattouw you want me to revert it? [23:10:47] Yes please [23:10:52] Still trying to find my test commit :/ [23:12:28] Ok [23:12:58] Hmm, I should build 2.12.4 and put it in a deb [23:13:05] ostriches yay [23:13:08] it will fix my bug [23:13:21] with it incorrectly saying the wrong user through the gerrit inline edit [23:13:46] Guess I have to rebuild buck though [23:13:53] Oh [23:13:54] * ostriches mutters something about losing all his dev work week before last [23:13:58] :( [23:14:14] ostriches why not setup a instance on labs to do it [23:14:20] that way you doint loose anything [23:14:27] Because labs instances are slow :p [23:15:00] Oh [23:15:17] ostriches i built it on labs and it was fast for me. [23:15:51] RoanKattouw could you open a bug please about fixing the cutting off the diff please? [23:16:04] My laptop is really really fast though :D [23:16:18] I think I did back then, trying to find it [23:16:22] I don't trust machines I can't see! [23:16:28] Clouds are scary! [23:16:29] Ok [23:16:45] ostriches lol, i should really see if it work on windows bash. [23:16:56] they finally fixed screen [23:17:22] Your running it on your p [23:17:23] pc [23:17:31] is it capable of running buck? [23:17:53] paladox: https://bugs.chromium.org/p/gerrit/issues/detail?id=4292 [23:17:58] Thanks [23:18:04] i mean a task in phabricator [23:18:25] RoanKattouw ^^ so we can track it too [23:18:27] please [23:18:42] Oh, I see [23:18:51] In what project? Is there a Gerrit-Upstream project or something? [23:19:28] RoanKattouw: [Gerrit] + [Upstream] [23:19:31] I think? [23:19:36] There's a column in my workboard [23:19:38] OK [23:19:38] Also, I think :) [23:19:40] Will do [23:20:16] Yippee, build fixed! [23:20:16] Project performance-webpagetest-wmf build #2468: 09FIXED in 1 hr 37 min: https://integration.wikimedia.org/ci/job/performance-webpagetest-wmf/2468/ [23:20:20] Thankyou [23:20:22] very much [23:20:25] and thankyou for reporting [23:20:40] im now testing to see the difference between gerrit-test and gerrit-new [23:22:01] 10Gerrit, 07Upstream: Gerrit's new side-by-side diff screen sometimes cuts off the last few characters of a line - https://phabricator.wikimedia.org/T144565#2603722 (10Catrope) [23:22:06] ostriches are you building gerrit 2.12.4 now? [23:22:47] paladox: Yeah I wanted to import my test change from gerrit-test to gerrit-new but I wasn't sure how. Are you working on that? [23:22:59] Yes [23:24:22] RoanKattouw http://gerrit-new.wmflabs.org/r/#/c/2/2/tests/fixtures/layout-cloner.yaml [23:24:29] nope still looks broken in gerrit 2.13 [23:25:44] paladox: Already done ;-) [23:25:48] Told you my computer was fast! [23:25:54] ostriches oh wow [23:25:58] I'll do the debian package stuff tomorrow. [23:26:01] I'm tired [23:26:05] Ok [23:26:06] :) [23:26:56] ostriches RoanKattouw https://gerrit.wikimedia.org/r/308095 [23:47:56] * paladox is going to watch tv now? [23:47:58] :) [23:57:00] but not if it is a anonnonmouse ip [23:57:34] go bt they enabled ipv6 [23:58:31] Yeah editing multi-paragraph comments is what was broken in IE [23:59:11] And one of the patches for it has been SWATted, but it won't be fixed until all 3 patches are SWATted [23:59:39] Ok [23:59:56] But it worked for editing a @paladox comment for example