[00:07:03] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL [00:07:19] thats new ^^ [00:07:26] never seen that before [00:07:33] ostriches twentyafterfour ^^ [00:10:08] oh boy, looks like: https://gerrit.wikimedia.org/r/#/c/304263/ [00:11:14] heh, a bit low :) [00:11:18] or, we have some errors [00:11:27] (I mean, yes, we do, I meant, "too many") [00:11:32] which is also true [00:11:35] whatever... [00:16:26] hrm, except I'm not seeing these errors. https://graphite-labs.wikimedia.org/render/?width=664&height=308&_salt=1479946556.073&target=logstash.rate.mediawiki.fatal.ERROR.sum&target=logstash.rate.mediawiki.exception.ERROR.sum&lineMode=connected&from=-10minutes&drawNullAsZero=true [00:17:27] hrmmm [00:18:41] oh, hi [00:18:45] i merged that [00:19:25] I saw that :) [00:20:04] seems to be jumping up to critical, but I don't see the errors actually happening in graphite, which must mean I did something wrong here :\ [00:23:22] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#2588690 (10greg) I'd like to make the ci-staging environment an explicit blocker for this, OK? (we can talk about it on Monday in our team meeting... [01:39:25] 10Beta-Cluster-Infrastructure, 05Goal, 13Patch-For-Review, 07Puppet: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#2819845 (10Krenair) So now we have: ```alex@alex-laptop:~/Development/Wikimedia/Operations-Puppet (production)$ git grep -E "class .*::beta" modules/mediawiki/ma... [04:55:10] (03PS1) 10Gergő Tisza: Add Huji (SecurePoll maintainer) to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/323362 [05:23:16] 10Beta-Cluster-Infrastructure, 05Goal, 13Patch-For-Review, 07Puppet: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#973295 (10demon) Just came across `beta::saltmaster::tools`. It doesn't even appear used... [05:26:40] 10Beta-Cluster-Infrastructure, 05Goal, 13Patch-For-Review, 07Puppet: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#2820019 (10Krenair) It's used on deployment-salt02: https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep/host/deployment-salt02 [05:26:56] 10Beta-Cluster-Infrastructure, 03Scap3: Rewrite wmf-beta-update-databases.py as plugin - https://phabricator.wikimedia.org/T151519#2820020 (10demon) [05:27:46] 10Beta-Cluster-Infrastructure, 05Goal, 13Patch-For-Review, 07Puppet: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#2820032 (10demon) Do we even need salt in beta? ;-) [05:29:11] 10Beta-Cluster-Infrastructure, 03Scap3: Rewrite wmf-beta-update-databases.py as plugin - https://phabricator.wikimedia.org/T151519#2820035 (10demon) (and related stuff like wmf-beta-autoupdate, wmf-beta-mwconfig-update) [05:31:38] 10Beta-Cluster-Infrastructure, 05Goal, 13Patch-For-Review, 07Puppet: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#2820037 (10Krenair) That's way out of the scope of this task, but yes, I have used it many times [05:39:08] Krenair: Salt is dumb [05:39:12] EOM [05:46:30] What did NaCl do to you? [05:47:12] Unspeakable horrors [08:04:31] (03PS2) 10Hashar: Add Huji (SecurePoll maintainer) to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/323362 (owner: 10Gergő Tisza) [08:05:11] (03CR) 10Hashar: [C: 032] "Annoyingly the email has to be added in two lists :( Amended :]" [integration/config] - 10https://gerrit.wikimedia.org/r/323362 (owner: 10Gergő Tisza) [08:05:53] (03Merged) 10jenkins-bot: Add Huji (SecurePoll maintainer) to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/323362 (owner: 10Gergő Tisza) [08:16:34] 10Continuous-Integration-Config, 13Patch-For-Review: Set up phpunit structure tests for MediaWiki skin repositories - https://phabricator.wikimedia.org/T68926#2820177 (10Paladox) @hashar we need to add support for skin dependacies including supporting skins depending on extensions and skins please? On eknown... [08:24:37] 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet, 07Technical-Debt, 07Tracking: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#2820182 (10hashar) [08:24:39] 10Beta-Cluster-Infrastructure, 05Goal, 13Patch-For-Review, 07Puppet: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#2820179 (10hashar) 05Open>03Resolved a:03hashar >>! In T86644#2820032, @demon wrote: > Do we even need salt in beta? ;-) When you get 40+ instances. Yes d... [08:29:02] 10Beta-Cluster-Infrastructure, 06Operations, 07Puppet, 07Technical-Debt, 07Tracking: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#2820188 (10Krenair) [08:29:04] 10Beta-Cluster-Infrastructure, 05Goal, 13Patch-For-Review, 07Puppet: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#2820185 (10Krenair) 05Resolved>03Open a:05hashar>03None This task is not complete. I listed 7 existing cases above. [08:30:45] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Jenkins: Upgrade Jenkins from 1.x to latest 2.x - https://phabricator.wikimedia.org/T144106#2820189 (10hashar) Yes that is implied. @thcipriani and I discussed about it last week and agreed the upgrade requires proper testing. Thus we nee... [08:32:21] 06Release-Engineering-Team, 06Labs: Request for CI staging project - https://phabricator.wikimedia.org/T150772#2820191 (10hashar) Labs project is `ci-staging` https://wikitech.wikimedia.org/wiki/Nova_Resource:Ci-staging [08:34:54] 10Beta-Cluster-Infrastructure, 05Goal, 13Patch-For-Review, 07Puppet: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#2820196 (10hashar) Sorry made a mistake when replying :D [08:43:42] 06Release-Engineering-Team, 10Phabricator: Import projects and tasks into phab-01 - https://phabricator.wikimedia.org/T151527#2820217 (10Paladox) [08:45:06] 06Release-Engineering-Team, 10Phabricator: Import projects and tasks into phab-01 - https://phabricator.wikimedia.org/T151527#2820217 (10Paladox) [08:46:28] hashar: o/ [08:47:36] 06Release-Engineering-Team, 10Wikimedia-Developer-Summit, 06Developer-Relations (Oct-Dec-2016), 07Documentation: Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2820249 (10Qgil) Last year the assignment of roles before the beginni... [08:47:51] I'd need to swap a cherry pick on the puppet master [08:48:44] so what I'd do is a git rebase -i to remove the previous patch, and then another git cherry pick to add the newer version (otherwise they conflicts) [08:49:18] is it doable and safe? :) [08:52:43] elukey: yeah [08:52:55] elukey: wanna do it with a screen sharing in hangouts? [08:53:08] can show you [08:53:10] reset back [08:53:14] and watch over your shoulders :D [08:55:14] hashar: should be easy like root + cd /var/lib/git/.../puppet + git rebase -i + dropping the line related to my commit + save [08:55:25] yeah [08:55:30] then cherry pick the fresh patch [08:55:56] all right let me try, should be safe even without supervision [08:56:02] there might be some puppet runs failling while you are doing the rebase or some inconsistency, but it is not much of a trouble [08:56:07] the next puppet run would catch up [08:58:11] done! [08:58:40] !log rebased puppet operations git repo on deployment-puppetmaster to refresh https://gerrit.wikimedia.org/r/#/c/322268/ [08:58:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:01:15] hashar: I figured out why the previous patch wasn't working, the RedirectMatch was looping over and over :P [09:06:35] 10Beta-Cluster-Infrastructure, 10Shinken, 13Patch-For-Review, 07Wikimedia-Incident, 07Wikimedia-log-errors: Shinken alert for beta error rate - https://phabricator.wikimedia.org/T141785#2512020 (10hashar) That is nice! https://gerrit.wikimedia.org/r/304263 managed to get an alarm overnight on beta-cluste... [09:07:23] !!! [09:26:48] 10Gerrit, 10Phabricator: Remove visible email addresses from emails from gerrit.wikimedia.org - https://phabricator.wikimedia.org/T151529#2820283 (10Billinghurst) [09:30:13] 10Gerrit, 07Upstream: Remove visible email addresses from emails from gerrit.wikimedia.org - https://phabricator.wikimedia.org/T151529#2820297 (10Peachey88) [09:41:31] 10Gerrit, 07Upstream: Remove visible email addresses from emails from gerrit.wikimedia.org - https://phabricator.wikimedia.org/T151529#2820283 (10Krenair) The commit log will contain most of these email addresses, and LDAP probably has the rest. It's not really possible to prevent this stuff being published in... [09:53:35] 10Gerrit, 07Upstream: Remove visible email addresses from emails from gerrit.wikimedia.org - https://phabricator.wikimedia.org/T151529#2820283 (10hashar) Gerrit is a code review system backed up by git. In git when you do a modification of code, you pack it in a commit which has metadata attached to it, namely... [10:09:35] 10Gerrit, 07Upstream: Remove visible email addresses from emails from gerrit.wikimedia.org - https://phabricator.wikimedia.org/T151529#2820433 (10Aklapper) 05Open>03declined Declining the request to "remove visible email addresses from emails from gerrit" as per previous comments. [10:13:12] hashar, you sure about "Gerrit verify that your account email matches the commit author email, else it rejects it"? [10:13:38] I've committed from wmflabs and wmnet addresses before and it's never taken issue at that [10:15:26] * Krenair has to go now but will be back in the evening [10:25:52] 10Gerrit, 07Upstream: Remove visible email addresses from emails from gerrit.wikimedia.org - https://phabricator.wikimedia.org/T151529#2820481 (10Billinghurst) Then please remove me from the gerrit review system. [10:29:46] 10Gerrit, 07Upstream: Remove visible email addresses from emails from gerrit.wikimedia.org - https://phabricator.wikimedia.org/T151529#2820495 (10Krenair) It may be difficult/impossible to do that without removing you from LDAP which can break all sorts of things, most obviously your labs access [10:33:56] 10Gerrit, 07Upstream: Remove visible email addresses from emails from gerrit.wikimedia.org - https://phabricator.wikimedia.org/T151529#2820512 (10Billinghurst) Then tell me how we are complying with the Wikimedia privacy policy. ``` We are committed to: - Describing how your information may be used or... [10:35:34] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Ensure chromedriver is installed (for Selenium) - https://phabricator.wikimedia.org/T117418#2820518 (10hashar) The chrome drivers are installed in Puppet via modules/contint/manifests/browsers.pp I have no idea why there are no links in /usr/bin, maybe we... [11:44:11] 10Gerrit, 07Upstream: Using Gerrit/git requires the email registered via wikitech and ends ups being voluntary disclosed (break of privacy?) - https://phabricator.wikimedia.org/T151529#2820682 (10hashar) 05declined>03Open p:05Triage>03Normal [11:49:35] 10Gerrit, 07Privacy, 07Upstream: Using Gerrit/git requires the email registered via wikitech and ends ups being voluntary disclosed (break of privacy?) - https://phabricator.wikimedia.org/T151529#2820695 (10hashar) I quickly talked to @Aklapper about it and rephrased the whole task so that instead of being a... [11:49:48] 10Gerrit, 07Privacy: Using Gerrit/git requires the email registered via wikitech and ends ups being voluntary disclosed (break of privacy?) - https://phabricator.wikimedia.org/T151529#2820697 (10hashar) [13:20:48] 10Gerrit: move mediawiki/rcsub.git under mediawiki/tools - https://phabricator.wikimedia.org/T49554#2820865 (10hashar) 05Open>03declined I have marked `mediawiki/rcsub.git` as an archived repository and made it read-only in Gerrit. [13:28:50] 10Gerrit, 07Privacy: Using Gerrit/git requires the email registered via wikitech and ends ups being voluntary disclosed (break of privacy?) - https://phabricator.wikimedia.org/T151529#2820877 (10Billinghurst) @hashar It is not even making a patch. Someone simply adding you into review a patch does it. [13:43:17] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: /etc/init.d/jenkins script provided by Debian doesn't work properly - https://phabricator.wikimedia.org/T53817#2820912 (10hashar) 05Open>03Resolved a:03hashar We have moved to Debian Jessie and the init script is now managed by systemd. So... [13:53:34] 10Gerrit: new extensions should be added to mediawiki/extensions.git automatically - https://phabricator.wikimedia.org/T48462#2820960 (10hashar) 05Open>03declined We do it manually from time to time using the sync-with-gerrit.py script at the root of mediawiki/extensions.git repository. That is good enough. [13:54:29] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Operations: setup a DB backed parser cache - https://phabricator.wikimedia.org/T55457#2820963 (10hashar) [14:10:51] 10Continuous-Integration-Infrastructure, 07Tracking: Have unit tests of all wmf deployed extensions pass when installed together, in both PHP-Zend and HHVM (tracking) - https://phabricator.wikimedia.org/T69216#2821119 (10hashar) [14:13:34] 10Deployment-Systems: Create phpunit test in mediawiki-config repo to validate Parsoid settings - https://phabricator.wikimedia.org/T70532#2821144 (10hashar) 05Open>03declined That is less of a problem nowadays. The services are defined at a single place via `wmfLocalServices` so it is less likely a configu... [15:01:58] 10Browser-Tests-Infrastructure, 07Ruby, 15User-zeljkofilipin: Mediawiki Ruby gem incorrectly assumes path to index.php - https://phabricator.wikimedia.org/T149169#2821250 (10zeljkofilipin) a:05zeljkofilipin>03None [15:37:08] 10Continuous-Integration-Infrastructure: Garbage collect orphan jobs workspaces on slaves - https://phabricator.wikimedia.org/T94408#2821319 (10hashar) 05Open>03Resolved That is less of a concern now that most jobs are running on disposable instances. As such there is less need to garbage collect workspaces. [15:43:42] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #239: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/239/ [15:44:26] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:46:07] !log removing https://gerrit.wikimedia.org/r/#/c/322268/ from the list of cherry picks on puppet master since it is not the right way to go [15:46:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:15:28] 10Gerrit: move mediawiki/rcsub.git under mediawiki/tools - https://phabricator.wikimedia.org/T49554#2821737 (10demon) 05declined>03Open >>! In T49554#2820865, @hashar wrote: > I have marked `mediawiki/rcsub.git` as an archived repository and made it read-only in Gerrit. Why? This task was not about doing t... [19:37:44] (03PS1) 10Gergő Tisza: [ZoomableImages] add endpoint [integration/config] - 10https://gerrit.wikimedia.org/r/323438 [19:38:27] (03PS3) 10Hashar: [SemanticImageInput] Add npm + ext dependency [integration/config] - 10https://gerrit.wikimedia.org/r/280764 (owner: 10Paladox) [19:39:23] (03CR) 10jenkins-bot: [V: 04-1] [ZoomableImages] add endpoint [integration/config] - 10https://gerrit.wikimedia.org/r/323438 (owner: 10Gergő Tisza) [19:39:58] (03CR) 10Hashar: [C: 032] [SemanticImageInput] Add npm + ext dependency [integration/config] - 10https://gerrit.wikimedia.org/r/280764 (owner: 10Paladox) [19:40:49] (03Merged) 10jenkins-bot: [SemanticImageInput] Add npm + ext dependency [integration/config] - 10https://gerrit.wikimedia.org/r/280764 (owner: 10Paladox) [19:48:49] 10Gerrit: move mediawiki/rcsub.git under mediawiki/tools - https://phabricator.wikimedia.org/T49554#2821975 (10hashar) 05Open>03Resolved a:03hashar Because that is a one time experiment with 8 commits, all done over two days in March 2013. Eg it is defacto abandoned and I dont see a point in keeping that... [19:50:58] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible, 07Easy, 13Patch-For-Review, and 2 others: "Connect to 'deployment.eqiad.wmnet' instead" when you ssh into deployment-tin on Beta - https://phabricator.wikimedia.org/T146505#2821982 (10hashar) 05Open>03Resolved Thanks @krenair and @akosiaris [20:08:48] 10Gerrit: move mediawiki/rcsub.git under mediawiki/tools - https://phabricator.wikimedia.org/T49554#2821998 (10demon) I didn't even realize this was your task that had been opened :P [20:10:46] 10Gerrit, 07Privacy: Using Gerrit/git requires the email registered via wikitech and ends ups being voluntary disclosed (break of privacy?) - https://phabricator.wikimedia.org/T151529#2822003 (10demon) p:05Normal>03High >>! In T151529#2820495, @Krenair wrote: > It may be difficult/impossible to do that wit... [20:11:14] 10Gerrit, 07Privacy: Using Gerrit/git requires the email registered via wikitech and ends ups being voluntary disclosed (break of privacy?) - https://phabricator.wikimedia.org/T151529#2822005 (10demon) a:03demon [20:19:25] 10Gerrit: Remove mediawiki/extensions/* repos that have never been used - https://phabricator.wikimedia.org/T115262#2822026 (10hashar) [20:20:40] 10Gerrit: Remove mediawiki/extensions/* repos that have never been used - https://phabricator.wikimedia.org/T115262#1719326 (10hashar) 05Open>03Resolved a:03hashar I have finally found time to review the list. Repos that had no change have been hidden. For the ones having changes I have abandoned them (mos... [20:49:59] !log make contint1001 Jenkins slave to only builds jobs with a label matching the node https://integration.wikimedia.org/ci/computer/contint1001/configure T86659 [20:50:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:52:00] 10Continuous-Integration-Infrastructure, 07Tracking: gallium and lanthanum disks full (tracking) - https://phabricator.wikimedia.org/T91211#2822050 (10hashar) [20:52:03] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Gerrit, 13Patch-For-Review, 07Technical-Debt: Disable Gerrit replication to production slaves - https://phabricator.wikimedia.org/T86661#2822051 (10hashar) [20:52:06] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 10releng-201415-Q3, 10releng-201415-Q4, and 2 others: [EPIC] Run CI jobs in disposable VMs - https://phabricator.wikimedia.org/T47499#2822052 (10hashar) [20:52:08] 10Continuous-Integration-Infrastructure: Migrate all jobs to labs slaves - https://phabricator.wikimedia.org/T86659#2822047 (10hashar) 05Open>03Resolved a:03hashar Paladox pointed: > jobs which have no node: stanza in JJB configuration file. Important I have changed [[ https://integration.wikimedia.org/ci... [20:52:49] 10Gerrit, 07Privacy: Using Gerrit/git requires the email registered via wikitech and ends ups being voluntary disclosed (break of privacy?) - https://phabricator.wikimedia.org/T151529#2822053 (10demon) a:05demon>03None I've marked your account inactive (so you can't login) and pruned all of the comments yo... [20:54:24] 10Gerrit, 13Patch-For-Review: Remove mediawiki/extensions/* repos that have never been used - https://phabricator.wikimedia.org/T115262#1719326 (10Nemo_bis) >>! In T115262#1733468, @Legoktm wrote: > Some of these repos have intial patches in gerrit that haven't been reviewed/merged, and shouldn't be deleted.... [20:55:47] 10Gerrit, 13Patch-For-Review: Remove mediawiki/extensions/* repos that have never been used - https://phabricator.wikimedia.org/T115262#2822058 (10demon) >>! In T115262#2822055, @Nemo_bis wrote: >>>! In T115262#1733468, @Legoktm wrote: >> Some of these repos have intial patches in gerrit that haven't been revi... [21:08:12] 10Gerrit: Gerrit upload-pack send ALL references causing massive network I/O on common operations - https://phabricator.wikimedia.org/T103990#2822105 (10hashar) From https://bugs.chromium.org/p/gerrit/issues/detail?id=175 > > With https://gerrit-review.googlesource.com/#/c/72258/, Gerrit 2.12 adopted JGit 4.1.... [21:59:46] 10Gerrit, 07Privacy: Using Gerrit/git requires the email registered via wikitech and ends ups being voluntary disclosed (break of privacy?) - https://phabricator.wikimedia.org/T151529#2822160 (10Krenair) >>! In T151529#2822053, @demon wrote: > I've marked your account inactive (so you can't login) and pruned a... [22:52:05] 10Gerrit, 07Privacy: Using Gerrit/git requires the email registered via wikitech and ends ups being voluntary disclosed (break of privacy?) - https://phabricator.wikimedia.org/T151529#2822185 (10demon) >>! In T151529#2822160, @Krenair wrote: >>>! In T151529#2822053, @demon wrote: >> I've marked your account in...