[00:22:46] 10Tool-Labs-tools-Xtools, 06Community-Tech: [Epic] Rewrite XTools: Top edits - https://phabricator.wikimedia.org/T160137#3224591 (10Samwilson) [00:22:47] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: XTools: Top edits - 'All' namespaces option - https://phabricator.wikimedia.org/T160721#3224589 (10Samwilson) 05Open>03Resolved This is now merged and deployed to the dev site. Note that the 'all' option is above 'article' in the namespace list, and the la... [01:40:21] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Xephyr826 was created, changed by Xephyr826 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Xephyr826 edit summary: Created page with "{{Tools Access Request |Justification=I intend to use the tools project to improve Tool Labs documentation. |Completed=false |User Name=Xephyr826 }}" [02:45:27] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Pages created interface gives error for most projects in new XTools - https://phabricator.wikimedia.org/T163634#3203727 (10MusikAnimal) 05Open>03Resolved [03:24:20] 06Labs, 10Tool-Labs: Add HHVM backend for webservice - https://phabricator.wikimedia.org/T164161#3224668 (10bd808) 05Open>03stalled p:05Triage>03Low I'm not having very good luck with experiments based on setting up HHVM in fastcgi mode automatically started by lighttpd. The problem is caused by [[http... [03:36:41] 06Labs, 10Tool-Labs: Add HHVM backend for webservice - https://phabricator.wikimedia.org/T164161#3224678 (10Amitie_10g) As HHVM in Proxygen mode worked properly, do you think is appropriate to document that at Wikitech as an experimental deployment? [03:56:15] 06Labs, 10Tool-Labs: Add HHVM backend for webservice - https://phabricator.wikimedia.org/T164161#3224681 (10Amitie_10g) For the record, I'm currently running the [[https://tools.wmflabs.org/webarchivebot|WebArchiveBOT's Web service]] as generic web service using the script provided by@bd808 since several hours... [03:59:33] 06Labs, 10Tool-Labs: Add HHVM backend for webservice - https://phabricator.wikimedia.org/T164161#3224682 (10bd808) >>! In T164161#3224678, @Amitie_10g wrote: > As HHVM in Proxygen mode worked properly, do you think is appropriate to document that at Wikitech as an experimental deployment (until the became offi... [04:06:16] 06Labs, 10Tool-Labs: Add HHVM backend for webservice - https://phabricator.wikimedia.org/T164161#3224683 (10Amitie_10g) I'll document that tomorrow. A good way to get debug information is using the logs, that I'll add to the configuration file. Also, should be a good idea to try HHVM in Daemon instead of the... [04:10:54] 06Labs, 10Tool-Labs: Add HHVM backend for webservice - https://phabricator.wikimedia.org/T164161#3224686 (10bd808) >>! In T164161#3224683, @Amitie_10g wrote: > I'll document that tomorrow. > > A good way to get debug information is using the logs, that I'll add to the configuration file. Also, should be a goo... [04:34:23] 06Labs, 10Tool-Labs: Add HHVM backend for webservice - https://phabricator.wikimedia.org/T164161#3224693 (10Amitie_10g) >>! In T164161#3224686, @bd808 wrote: >>>! In T164161#3224683, @Amitie_10g wrote: >> I'll document that tomorrow. >> >> A good way to get debug information is using the logs, that I'll add t... [04:41:10] (03CR) 10BryanDavis: [C: 032] Add basic suppression for membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/348388 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [04:42:09] (03CR) 10BryanDavis: [C: 032] Add links to user's accounts to membership request [labs/striker] - 10https://gerrit.wikimedia.org/r/348386 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [04:42:42] (03CR) 10BryanDavis: [C: 032] Add link to membership requests list [labs/striker] - 10https://gerrit.wikimedia.org/r/348387 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [04:43:05] (03CR) 10BryanDavis: [C: 032] Fix use of deprecated Logger.warn() method [labs/striker] - 10https://gerrit.wikimedia.org/r/350630 (owner: 10BryanDavis) [04:43:21] (03Merged) 10jenkins-bot: Add links to user's accounts to membership request [labs/striker] - 10https://gerrit.wikimedia.org/r/348386 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [04:43:41] (03CR) 10BryanDavis: [C: 032] Switch logstash transport from UDP to TCP [labs/striker] - 10https://gerrit.wikimedia.org/r/350638 (https://phabricator.wikimedia.org/T151422) (owner: 10BryanDavis) [05:18:51] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:34:54] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#3224704 (10Xephyr826) Thanks @zhuyifei1999 ! I started the tutorial here: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/My_first_Pywiki... [05:58:50] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [06:51:55] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:26:56] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [09:10:18] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#3224750 (10zhuyifei1999) @Xephyr826 See also https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Developing#Pywikibot [09:31:30] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Psubhashish was created, changed by Psubhashish link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Psubhashish edit summary: Created page with "{{Tools Access Request |Justification=I would like to make a converter that a fellow Wikimedian User:Prateek_Pattanaik and I are building that converts the Odia language into..." [10:21:47] 10Tool-Labs-tools-Pageviews: Add SPARQL as input for Massviews - https://phabricator.wikimedia.org/T164181#3224802 (10Aklapper) [10:57:53] 10Tool-Labs-tools-Pageviews, 13Patch-For-Review: Add ability to query for legacy pageviews for projects - https://phabricator.wikimedia.org/T149358#2749796 (10Nemo_bis) >>! In T149358#3141396, @Nuria wrote: > @MusikAnimal we still to triple vet data and fix docs, we will send an announcement when all this work... [12:34:30] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: XTools: Top edits - 'All' namespaces option - https://phabricator.wikimedia.org/T160721#3224900 (10Stevietheman) @Samwilson I ran 'All' for my own account, and all the entries looked familiar and highly likely to be correct. [12:39:36] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#3224905 (10chasemp) [12:39:38] 06Labs: Request creation of Wikidata Concepts labs project - https://phabricator.wikimedia.org/T163672#3224903 (10chasemp) 05Open>03stalled p:05Triage>03Low [14:03:28] 06Labs, 10MediaWiki-API: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3225024 (10MarcoAurelio) [14:36:57] 06Labs, 06Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static and develop procedures to keep it maintained - https://phabricator.wikimedia.org/T163721#3225107 (10Andrew) The icinga alert warned me about slippage between 1.28.1 and 1.28.2. I updated wikitech-static, and now the... [14:47:32] 10Tool-Labs-tools-Pageviews, 13Patch-For-Review: Add ability to query for legacy pageviews for projects - https://phabricator.wikimedia.org/T149358#3225135 (10Nuria) Indeed, this work is done and legacy pageviews are visible here: https://analytics.wikimedia.org/dashboards/reportcard/#pagecounts-dec-2007-dec-2... [15:15:04] 06Labs, 10Labs-Infrastructure: Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3225215 (10chasemp) >>! In T163796#3217927, @Andrew wrote: > Worst case scenario: What if every one of those instances suddenly partitioned and filled every bit of allocated space? On average, consumed spa... [15:30:31] 10PAWS: R kernel not available for PAWS - https://phabricator.wikimedia.org/T164198#3225260 (10Halfak) [16:31:13] 10Tool-Labs-tools-stewardbots: Let stewardbot display temporary rightchanges too - https://phabricator.wikimedia.org/T164204#3225412 (10Luke081515) [16:34:16] (03PS3) 10BryanDavis: Add link to membership requests list [labs/striker] - 10https://gerrit.wikimedia.org/r/348387 (https://phabricator.wikimedia.org/T162508) [16:37:07] (03CR) 10BryanDavis: Add link to membership requests list [labs/striker] - 10https://gerrit.wikimedia.org/r/348387 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [16:37:09] (03CR) 10BryanDavis: [C: 032] Add link to membership requests list [labs/striker] - 10https://gerrit.wikimedia.org/r/348387 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [16:38:22] (03Merged) 10jenkins-bot: Add link to membership requests list [labs/striker] - 10https://gerrit.wikimedia.org/r/348387 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [16:39:46] 10Tool-Labs-tools-Pageviews, 13Patch-For-Review: Add ability to query for legacy pageviews for projects - https://phabricator.wikimedia.org/T149358#3225464 (10MusikAnimal) I have a branch cut to add this to Siteviews, but there are still some major challenges. This is because people want the pagecounts alongsi... [16:41:03] (03PS3) 10BryanDavis: Add basic suppression for membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/348388 (https://phabricator.wikimedia.org/T162508) [16:41:05] (03PS3) 10BryanDavis: Notify admins of new membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/348389 (https://phabricator.wikimedia.org/T162508) [16:41:07] (03PS3) 10BryanDavis: Send tool maintainers a notice when a git repo is created [labs/striker] - 10https://gerrit.wikimedia.org/r/348390 [16:41:09] (03PS2) 10BryanDavis: Fix use of deprecated Logger.warn() method [labs/striker] - 10https://gerrit.wikimedia.org/r/350630 [16:41:19] (03PS2) 10BryanDavis: Switch logstash transport from UDP to TCP [labs/striker] - 10https://gerrit.wikimedia.org/r/350638 (https://phabricator.wikimedia.org/T151422) [16:42:45] (03CR) 10BryanDavis: Add basic suppression for membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/348388 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [16:42:47] (03CR) 10BryanDavis: [C: 032] Add basic suppression for membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/348388 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [16:43:00] (03CR) 10BryanDavis: [C: 032] Notify admins of new membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/348389 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [16:43:50] (03Merged) 10jenkins-bot: Add basic suppression for membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/348388 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [16:44:05] (03Merged) 10jenkins-bot: Notify admins of new membership requests [labs/striker] - 10https://gerrit.wikimedia.org/r/348389 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [16:44:07] (03Merged) 10jenkins-bot: Send tool maintainers a notice when a git repo is created [labs/striker] - 10https://gerrit.wikimedia.org/r/348390 (owner: 10BryanDavis) [16:44:09] (03Merged) 10jenkins-bot: Fix use of deprecated Logger.warn() method [labs/striker] - 10https://gerrit.wikimedia.org/r/350630 (owner: 10BryanDavis) [16:44:36] (03Merged) 10jenkins-bot: Switch logstash transport from UDP to TCP [labs/striker] - 10https://gerrit.wikimedia.org/r/350638 (https://phabricator.wikimedia.org/T151422) (owner: 10BryanDavis) [17:13:52] 10Tool-Labs-tools-Xtools, 06Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3225541 (10Matthewrbowker) [17:13:53] 10Tool-Labs-tools-Xtools, 06Community-Tech: Resolve timeout errors for Adminscore - https://phabricator.wikimedia.org/T164094#3225538 (10Matthewrbowker) 05Open>03Resolved a:05Matthewrbowker>03Superyetkin Thanks to @Superyetkin this task has been fixed. [17:14:07] 10Tool-Labs-tools-Pageviews: Add SPARQL as input for Massviews - https://phabricator.wikimedia.org/T164181#3225542 (10MusikAnimal) @Edgars2007 For starters, I am myself new to SPARQL but I've been wanting to get into it, so this sounds like a good opportunity to get my feet wet :) Regarding your third bullet po... [17:18:03] (03PS1) 10BryanDavis: Update for Tool Labs membership application and processing [labs/striker/staticfiles] - 10https://gerrit.wikimedia.org/r/351164 (https://phabricator.wikimedia.org/T162508) [17:18:44] (03CR) 10BryanDavis: [C: 032] Update for Tool Labs membership application and processing [labs/striker/staticfiles] - 10https://gerrit.wikimedia.org/r/351164 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [17:18:50] (03Merged) 10jenkins-bot: Update for Tool Labs membership application and processing [labs/striker/staticfiles] - 10https://gerrit.wikimedia.org/r/351164 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [17:22:11] 10Tool-Labs-tools-stewardbots: Let stewardbot display temporary rightchanges too - https://phabricator.wikimedia.org/T164204#3225569 (10MarcoAurelio) 05Open>03stalled p:05Triage>03Low StewardBot parses the log using a regex which detects which permissions got added and removed. If we were to implement th... [17:41:08] 10Tool-Labs-tools-Pageviews, 13Patch-For-Review: Add ability to query for legacy pageviews for projects - https://phabricator.wikimedia.org/T149358#3225622 (10Nuria) >I have a branch cut to add this to Siteviews, but there are still some major challenges. This is because people want the pagecounts alongside th... [17:41:36] I want to programmatically read the deletion logs of a few tens of thousands of pages on the English Wikipedia to search for keywords. Can Pywikibot do this? Can I download (just) the deletion logs either for the entire wiki or the pages I'm interested in? Can I get them from the database? I have wmflabs access. [17:43:57] codeofdusk, there are several options, depending on what you need [17:44:20] if it is a one time thing, if you just need to see a stream of changes, etc. [17:45:03] there is http://deletionpedia.org/en/Main_Page [17:45:20] jynus: One-time analysis of 24,316 pages. [17:45:28] just wondering how they do it, but https://github.com/guaka/deletionpedia [17:45:54] mutante: Nope, don't need to see deleted revisions. [17:46:09] mutante: Just the reasons pages were deleted (copyright violation, etc). [17:46:20] mutante: I know copyright violations are revision deleted now, but this is fairly old history. [17:46:31] ah! ok. got it. i had just seen "antidelete.py is based on pywikibot" [17:47:52] mutante: What's antidelete.py? [17:48:02] codeofdusk, https://dumps.wikimedia.org/enwiki/20170420/enwiki-20170420-pages-logging.xml.gz [17:48:20] or just go to https://dumps.wikimedia.org/enwiki/ [17:48:27] you do not need to download that for labs [17:48:30] it is available locally [17:48:50] codeofdusk: the script that they use to save all deleted pages, apparently. but see what jynus said, i misunderstood what you needed [17:48:51] jynus: Where? [17:49:05] so I would take that file and work on it, probably much easier than the api or the database [17:49:48] or you can pre-process that files to obtain only the deletes/pages you want, and generate a smaller file7db [17:50:10] codeofdusk, if you mean where it is locally, I am not sure- it is on the docs [17:50:17] let me search [17:51:03] check here: https://wikitech.wikimedia.org/wiki/Help:Shared_storage#.2Fpublic.2Fdumps [17:52:31] codeofdusk ^ [17:53:14] the big gotcha for using dumps on Labs/Tool Labs is that you should not try to uncompress them. There are some scripts around that can read the compressed dumps directly. [17:53:20] * bd808 looks for links [17:53:23] correct [17:53:31] just read them on the fly [17:53:43] jynus: How? [17:53:59] https://github.com/mediawiki-utilities/python-mwxml [17:54:29] bd808: Looking. [17:55:15] codeofdusk: halfak may be able to give you other tips or links to documentation on dumps processing [17:55:29] ! [17:55:29] hello :) [17:55:32] zcat ? [17:56:13] codeofdusk, here's an example of how to process XML dumps: https://github.com/mediawiki-utilities/python-mwxml/blob/master/ipython/labs_example.ipynb [17:56:35] This example uses mwxml and processed the Dutch Wikipedia for image-link insertion events. [17:57:14] I'm just about to jump into another meeting, but I'll be around in about an hour. [17:57:49] See http://pythonhosted.org/mwxml/ for basic docs. [17:58:16] (03PS1) 10BryanDavis: Add wheels for django-notifications-hq & python-keystoneclient [labs/striker/wheels] - 10https://gerrit.wikimedia.org/r/351170 (https://phabricator.wikimedia.org/T162508) [17:59:22] (03CR) 10BryanDavis: [C: 032] Add wheels for django-notifications-hq & python-keystoneclient [labs/striker/wheels] - 10https://gerrit.wikimedia.org/r/351170 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [17:59:28] (03Merged) 10jenkins-bot: Add wheels for django-notifications-hq & python-keystoneclient [labs/striker/wheels] - 10https://gerrit.wikimedia.org/r/351170 (https://phabricator.wikimedia.org/T162508) (owner: 10BryanDavis) [17:59:44] halfak: Thanks. Will look into this! [18:03:13] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1419 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:06:54] I can't import mwxml on labs. [18:12:26] Also, I want to process the logs, which appear to be in a different format than the main wiki dumps... [18:12:33] Disconnecting will be back in a minute. [18:18:13] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [18:34:30] Can the Python mwxml library read the pages-logging.xml.gz dumps on wmflabs? How do I get mwxml on labs ("import mwxml" gives a traceback)? [18:52:19] codeofdusk: for installing, make a virtual environment and use pip to install it there. [18:54:01] something like: virtualenv $HOME/venv; source $HOME/venv/bin/activate; pip install mwxml [18:56:02] bd808: Alright. [18:56:16] bd808: But the logging dump is only like 3GB, might be easier to just download locally. [18:56:28] bd808: Do you know if mwxml can handle the logging dumps? [18:57:10] I don't. I've actually never used the lib myself. [19:04:01] (03PS1) 10BryanDavis: [WIP] Implement Tool Labs membership application and processing [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/351175 (https://phabricator.wikimedia.org/T162508) [19:36:03] 06Labs, 10Labs-Infrastructure: Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3226039 (10Andrew) From nova: - labvirt - uuid - instance allocated storage - instance used storage from puppet config: - [y/n] for has provisioned more than initial partition spreadsheet calcs: - instance... [19:54:37] (03PS2) 10BryanDavis: Implement Tool Labs membership application and processing [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/351175 (https://phabricator.wikimedia.org/T162508) [20:14:27] 06Labs, 10wikitech.wikimedia.org, 05codfw-rollout: VisualEditor broken on wikitech: "Error loading data from server: apierror-visualeditor-docserver-http: HTTP 500." - https://phabricator.wikimedia.org/T163438#3226123 (10Jdforrester-WMF) [20:14:36] 06Labs, 10wikitech.wikimedia.org, 05codfw-rollout: VisualEditor broken on wikitech when codfw is primary: "Error loading data from server: apierror-visualeditor-docserver-http: HTTP 500." - https://phabricator.wikimedia.org/T163438#3197386 (10Jdforrester-WMF) [20:19:44] Hello, Amitie_10g here [20:19:54] I have a question about HHVM [20:21:27] Amitie_10g: just ask someone will be around to answer you shortly (I would execpt HHVM is something I don't know enough about to help you with) [20:22:31] As I already discussed that at Phabricator and Brian Davis offered an script to run HHVM in Proxygen mode, the Typechecker is urrently running in the Grid or not? [20:23:14] I ask that in order to properly document that at Wikitech [20:23:47] bd808: would you be able to answer that ^ considering you interacted with Amitie_10g about this in the past. [20:26:26] My question, more specific, is currently hh_server and hh_client running as service in the Grid or not? [20:57:19] 06Labs, 10MediaWiki-API: Tired of APIError: readonly - https://phabricator.wikimedia.org/T164191#3225024 (10Legoktm) The error message means that the eswiki db is lagged, yes. I think the main problem that you're having is that whatever script you are running isn't retrying after the readonly error, so you hav... [21:04:35] If Amitie_10g comes back around, no we aren't running typechecker/hh_server/hh_client as shared services in Tool Labs. Which seem sort of obvious since we have just barely hacked a kind of working HHVM setup over the weekend in T164161 [21:04:36] T164161: Add HHVM backend for webservice - https://phabricator.wikimedia.org/T164161 [21:15:50] (03PS1) 10Mholloway: Ensure that the Jenkins build timestamp is interpreted as UTC [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/351200 [21:18:17] PROBLEM - Puppet errors on tools-exec-1432 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:18:37] 06Labs, 10Labs-Infrastructure: Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3226297 (10chasemp) Thanks @andrew, I'm trying make sense of this :) I think we can reason backwards from the `disk_available_least` value for some things on allocated vs available. It seems like nova sho... [21:21:53] !log striker Deleting mw-vagrant managed VM on striker-deploy03 to rebuild it as a jessie VM [21:21:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL [21:28:40] 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-GettingStarted, 13Patch-For-Review: Labs undefined index due to data center switchover - https://phabricator.wikimedia.org/T163514#3226329 (10Mattflaschen-WMF) 05Open>03Resolved a:03Mattflaschen-WMF I checked earlier, and I don't think the above p... [21:28:58] Interesting read.- https://yahooeng.tumblr.com/post/159795571841/operating-openstack-at-scale [21:34:31] thanks Krinkle [21:58:18] RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [21:59:25] 06Labs, 06Operations, 10hardware-requests: Eqiad: (2) hardware access request for labnet1003/1004 - https://phabricator.wikimedia.org/T158204#3226404 (10chasemp) [22:58:10] !log striker Trying to revert from role::puppet::self to labs puppetmaster [22:58:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL [23:08:49] 06Labs, 06Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static and develop procedures to keep it maintained - https://phabricator.wikimedia.org/T163721#3226606 (10Dzahn) We have "wikitech-static CRIT - wikitech and wikitech-static out of sync (426228s > 200000s)" alerts on silve... [23:10:04] 06Labs, 10Labs-Infrastructure, 06Operations, 10Wikimedia-Apache-configuration, and 2 others: wikitech-static sync broken - https://phabricator.wikimedia.org/T101803#1347928 (10Dzahn) 05Resolved>03Open re-using the ticket again for the same issue. we have an alert https://icinga.wikimedia.org/cgi-bin/i... [23:10:18] 06Labs, 10Labs-Infrastructure, 06Operations, 10Wikimedia-Apache-configuration, and 2 others: wikitech-static sync broken - https://phabricator.wikimedia.org/T101803#3226610 (10Dzahn) p:05High>03Normal [23:32:08] !log striker Reverting to labs puppetmaster actually worked, but now I remember why there was a project puppetmaster :/ [23:32:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL