[00:47:32] 10Beta-Cluster-Infrastructure: New wiki cluster wikipedia indonesian language - https://phabricator.wikimedia.org/T143557#2905574 (10Mbrt) @Aklapper Mkay I can use enwiki but can you give me crat for testing [02:38:36] 10Gerrit, 06Release-Engineering-Team, 06Operations, 05Security, 07Upstream: Gerrit: Restarting gerrit could lead to data loss + accounts - https://phabricator.wikimedia.org/T154205#2905700 (10Paladox) @Aklapper i haven't tried yet reproducing data loss for accounts, but i managed with patches without eve... [02:50:44] 10Gerrit, 06Release-Engineering-Team, 06Operations, 07Upstream: Gerrit: Restarting gerrit could lead to data loss + accounts - https://phabricator.wikimedia.org/T154205#2905703 (10Peachey88) Do we actually have account_data that can be lost (and cause issues)? since all our account data is synced from a ex... [02:54:40] 10Gerrit, 06Release-Engineering-Team, 06Operations, 07Upstream: Gerrit: Restarting gerrit could lead to data loss + accounts - https://phabricator.wikimedia.org/T154205#2905707 (10Paladox) @Peachey88 even though we use ldap, gerrit stores all accounts in the index. Which is what caused T152640 that problem... [06:34:40] Project selenium-Wikibase » chrome,test,Linux,contintLabsSlave && UbuntuTrusty build #222: 04FAILURE in 1 hr 54 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/222/ [15:27:31] Hi all, I'd like to have 329597 deployed. This is a throttle rule for tomorrow event and the ticket was created in the UTC morning. See T154278 for details. [15:27:31] T154278: Temporary IP Cap Lift on mai.wiki - https://phabricator.wikimedia.org/T154278 [15:27:39] Is it possible? [15:27:50] ping greg-g [15:29:22] Urbanecm: hi you may want to send him an email as it is unlikly he will see your ping as it is christmas holidays, he may check his emails. [15:31:14] paladox, I've sent and e-mail already (after a while after patch creating). But I'm trying to get him (or anybody who can approve emergency deploy) by using more ways because I don't know what is the best (i.e. what is watched). [15:31:18] *an [15:31:35] Oh [15:31:52] But according to whois he didn't post anything in last 139 hours. [15:32:44] Oh, i guess ask in -operations to see if he can be contacted by phone if it is really important? [15:33:24] Wikimedia should be back from holidays on the 30 of december (not sure if that means they will get that day of too but thats what i herd from some other users) [15:33:38] Urbanecm ^^ [15:35:06] paladox, The event is held at December 30, 03:00 UTC. I don't know what mean "important" in this situation. [15:35:47] Oh [15:36:20] (just a note, I live in UTC+1 so I won't be available before the event but after the midnight) [15:37:57] ok [15:39:38] all of releng is out as far as i know so not sure what needs to be done to get permision to deploy the emergancy change. Maybe -operations would know what to do in a situration like this? Or even know how to get in contact with him. [15:39:49] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:39:56] Urbanecm ^^ [15:40:50] I always asked in -operations but one time I was directed here so I tried it here... Going to ask at -operations. [15:41:32] oh [15:44:07] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #275: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/275/ [15:48:29] paladox, as you probably saw I've asked :) [15:48:42] Yep, thanks :) [15:48:48] yw [15:49:13] normaly -releng would be the place to ask but as releng is out you would be unlikly to get a reply from them in here. [16:51:04] 10Gerrit, 06Release-Engineering-Team, 06Operations, 07Upstream: Gerrit: Restarting gerrit could lead to data loss + accounts - https://phabricator.wikimedia.org/T154205#2906496 (10Paladox) This is the commit that broke it https://gerrit-review.googlesource.com/#/c/86804/ we can revert it and apply it loca... [16:52:48] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:54:51] 10Gerrit, 10BlueSpice, 07Upstream: Merge/Submit error on Gerrit: "org.eclipse.jgit.errors.MissingObjectException: Missing unknown" for BlueSpiceExtensions' REL1_27 branch - https://phabricator.wikimedia.org/T153079#2906497 (10Paladox) @Osnard I'm wondering did this problem start happening on wednesday the 7t... [17:01:38] 10Gerrit, 06Release-Engineering-Team, 06Operations, 07Upstream: Gerrit: Restarting gerrit could lead to data loss + accounts - https://phabricator.wikimedia.org/T154205#2906498 (10Paladox) it will require an index after we upgrade gerrit to include ^^ just so we doint lose any commits. [17:27:47] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [19:04:48] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests, 13Patch-For-Review: On beta metawiki, a mix of the beta enwiki and the production metawiki logos show - https://phabricator.wikimedia.org/T125942#2906576 (10Dereckson) a:03tomasz [19:06:19] 10Gerrit, 10BlueSpice, 07Upstream: Merge/Submit error on Gerrit: "org.eclipse.jgit.errors.MissingObjectException: Missing unknown" for BlueSpiceExtensions' REL1_27 branch - https://phabricator.wikimedia.org/T153079#2906579 (10Mglaser) @Paladox The last successful merge was on December 9th for BlueSpiceFounda... [19:39:55] 10Gerrit, 10BlueSpice, 07Upstream: Merge/Submit error on Gerrit: "org.eclipse.jgit.errors.MissingObjectException: Missing unknown" for BlueSpiceExtensions' REL1_27 branch - https://phabricator.wikimedia.org/T153079#2906645 (10Paladox) Thank you for the link and info @Mglaser looking at the last successful me... [19:48:01] 10Gerrit, 10BlueSpice, 07Upstream: Merge/Submit error on Gerrit: "org.eclipse.jgit.errors.MissingObjectException: Missing unknown" for BlueSpiceExtensions' REL1_27 branch - https://phabricator.wikimedia.org/T153079#2906658 (10Paladox) Ah ha, cherry pick seems to be using parent commit i.e. cherry pick master... [19:55:16] 10Gerrit, 10BlueSpice, 07Upstream: Merge/Submit error on Gerrit: "org.eclipse.jgit.errors.MissingObjectException: Missing unknown" for BlueSpiceExtensions' REL1_27 branch - https://phabricator.wikimedia.org/T153079#2906659 (10Paladox) @Mglaser or @Osnard could you try re c+2 for https://gerrit.wikimedia.org/... [20:05:55] 10Gerrit, 10BlueSpice, 07Upstream: Merge/Submit error on Gerrit: "org.eclipse.jgit.errors.MissingObjectException: Missing unknown" for BlueSpiceExtensions' REL1_27 branch - https://phabricator.wikimedia.org/T153079#2906669 (10Paladox) Oh, i think i know where the problem came from, since when we first upgrad... [20:11:41] 10Gerrit, 10BlueSpice, 07Upstream: Merge/Submit error on Gerrit: "org.eclipse.jgit.errors.MissingObjectException: Missing unknown" for BlueSpiceExtensions' REL1_27 branch - https://phabricator.wikimedia.org/T153079#2906673 (10Paladox) But also it could have been gc too when it broke things. [20:37:39] 10Gerrit, 10BlueSpice, 07Upstream: Merge/Submit error on Gerrit: "org.eclipse.jgit.errors.MissingObjectException: Missing unknown" for BlueSpiceExtensions' REL1_27 branch - https://phabricator.wikimedia.org/T153079#2906712 (10Paladox) Yes this seems more like reindex as it seems that if you merge a change, r... [20:41:39] 10Gerrit, 06Release-Engineering-Team, 06Operations, 07Upstream: Gerrit: Restarting gerrit could lead to data loss + accounts - https://phabricator.wikimedia.org/T154205#2906714 (10Paladox) this https://gerrit-review.googlesource.com/#/c/86804/ also broke it where you merge a change, restart gerrit, look a... [22:31:20] 10Deployment-Systems, 07I18n: Make LocalisationUpdate run daily again on Wikimedia wikis - https://phabricator.wikimedia.org/T154302#2906755 (10Nemo_bis) [22:31:43] 10Deployment-Systems, 07I18n: Make LocalisationUpdate run daily again on Wikimedia wikis - https://phabricator.wikimedia.org/T154302#2906769 (10Nemo_bis) p:05Triage>03High [22:38:23] 10Deployment-Systems, 07I18n: Make LocalisationUpdate run daily again on Wikimedia wikis - https://phabricator.wikimedia.org/T154302#2906775 (10Nemo_bis) The latest mention I see in https://wikitech.wikimedia.org/wiki/Server_Admin_Log is on 2016-12-22: > 22:11 thcipriani: disable l10nupdate cron for deployment... [22:43:03] 10Deployment-Systems, 07I18n: Make LocalisationUpdate run daily again on Wikimedia wikis - https://phabricator.wikimedia.org/T154302#2906755 (10greg) Right, it was disabled for the end of year code freeze. The change: https://gerrit.wikimedia.org/r/#/c/328738/ With the revert ready to go on Jan 3rd: https://g... [22:47:46] 10Deployment-Systems, 07I18n: Make LocalisationUpdate run daily again on Wikimedia wikis - https://phabricator.wikimedia.org/T154302#2906804 (10demon) p:05High>03Normal Not a high priority, this was planned. [23:02:26] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]