[01:00:26] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL - Socket timeout after 10 seconds [01:05:21] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 771123 bytes in 3.524 second response time [01:48:12] Is it possible to override https://github.com/wikimedia/operations-puppet/blob/production/modules/ssh/manifests/server.pp#L2 [01:55:36] nm [01:56:52] gah [02:02:57] how do I keep role::puppet::self from overwriting all my work every time I logout [02:24:23] puppetmaster_autoupdate is set false [10:22:09] 10Quarry: SELECT command denied to user for table 'logging_userindex' - https://phabricator.wikimedia.org/T99414#1291580 (10Nemo_bis) 3NEW [11:21:53] 10Tool-Labs: /etc/cron.daily/logrotate: gzip: stdin: file size changed while zipping - https://phabricator.wikimedia.org/T96007#1291620 (10valhallasw) I've tried to find the underlying cause for this, but the cron mail is not overly informative. Unfortunately, it also doesn't seem to happen on the same host ever... [14:01:27] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Hargup was created, changed by Hargup link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Hargup edit summary: Created page with "{{Tools Access Request |Justification=Hi, I plan to use the Tools project for the Wikipedia Gender Index project [1]. [1]: https://meta.wikimedia.org/wiki/Grants:IdeaLab/WI..." [14:02:59] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Hargup was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=159588 edit summary: [14:25:36] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ankitaa.bhowmick was created, changed by Ankitaa.bhowmick link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Ankitaa.bhowmick edit summary: Created page with "{{Tools Access Request |Justification=Component(Software architecture) Documentation maintenance bot |Completed=false |User Name=Ankitaa.bhowmick }}" [14:32:24] [13intuition] 15ChameleonWiki opened pull request #44: Remove Norwegian (no). Fallback "no" to "nb" (06master...06nono) 02https://github.com/Krinkle/intuition/pull/44 [14:55:01] Hi I created a new tool named "wigi". To become the tool account I did `become wigi` but I get the following message "You were added to the group tools.wigi after you started this login session. [14:55:02] You need to log out and in again to be able to "become wigi"." But logging out and then logging again is not helping. [14:55:38] hargup: let me take a look [14:57:06] hargup: odd. Did you fully close the ssh session? [14:59:05] First I did Ctrl-Z, then I realized that it might not fully close the session so I ran the `logout` command [14:59:15] `logout` after logging in again [14:59:55] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ankitaa.bhowmick was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=159594 edit summary: [15:00:38] hargup: sorry, that isn't really clear to me. Are you on linux? then run logout as often until you get back to your local shell, then run ssh again [15:00:44] hargup: on windows, close putty and fully reconnect [15:02:50] hargup: when I switch to your user, then run 'become wigi', it works without issues. [15:03:04] hargup: although that might be because my admin status somehow leaks through? not sure. [15:05:32] Thanks for the help, it works now. [15:05:43] I configured my ssh to cache the connections [15:06:16] hence it wasn't properly logging out, I just had to remove that socket file in .ssh [15:07:24] ah, that makes sense. [15:13:55] puppetmaster_autoupdate seems to be sticky on one of my instances. It continues to overwrite everything without a care in the world [15:14:39] I searched /etc/cron for the rebase job but couldn't find anything [15:20:59] Negative24: https://github.com/wikimedia/operations-puppet/blob/21c72942dd7bf25dbe0759d2f867082e966bfb45/modules/puppetmaster/manifests/gitsync.pp [15:21:16] so there should be a rebase_operations_puppet cron job somewhere [15:21:52] valhallasw: exactly. Can't find it. [15:22:00] but I'd say it's a bug that cronjob is not removed when puppetmaster_autoupdate is removed :/ [15:22:16] Negative24: let me check on the toolsbeta puppetmaster... [15:24:25] Negative24: check crontab -l as root? [15:24:40] because apart from /etc/cron.d, there's also the crontab... [15:26:46] valhallasw: oh there it is [15:27:50] I always get caught off guard when I remove a puppet class from my instance but things don't 'revert' [15:29:08] What is the recommended way to get changes to and from a self puppetmaster. The update script says that local hacking isn't good [15:32:08] Negative24: a local git checkout, then push it there [15:32:55] [remote "puppetmaster3"] [15:32:55] url = toolsbeta-puppetmaster3.eqiad.wmflabs:/var/lib/git/operations/puppet [15:33:26] and then there's something with permissions on that directory. let me see if I remember [15:34:04] git push --receive-pack="sudo git-receive-pack" -f puppetmaster3 HEAD:test-commit [15:34:12] then check out test-commit on the puppet master [15:35:02] ah ok [15:37:04] valhallasw: since /var/lib/git/operations/puppet is not a bare repository it wouldn't be able to be pushed to [15:37:18] Negative24: that's not true [15:37:21] ...wat? [15:37:38] Negative24: it just won't update the checkout automatically [15:37:46] you can push to non-bare repositories [15:39:44] I don't know. I'm just reading https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#Set_up_a_remote_machine_for_pushing_to_labs [15:42:10] What if I just chown /var/lib/git/operations? [15:43:11] dunno, might just work [15:51:34] 10Quarry: Suggested Feature: Put title of the query into the HTML title for page - https://phabricator.wikimedia.org/T99449#1292025 (10RaymondYee) 3NEW [15:55:55] [13intuition] 15Krinkle pushed 1 new commit to 06master: 02https://github.com/Krinkle/intuition/commit/da426e866f1645056aacbcf442221833719f9862 [15:55:55] 13intuition/06master 14da426e8 15Timo Tijhof: Convert fallbacks to JSON and update from latest MediaWiki master... [16:23:45] [13intuition] 15Krinkle pushed 1 new commit to 06master: 02https://github.com/Krinkle/intuition/commit/4bb1dbc25d7d0ce924c8c27dae59ff52351d8ea6 [16:23:46] 13intuition/06master 144bb1dbc 15Timo Tijhof: Convert Rtl.php to JSON and update from latest MediaWiki master... [16:28:28] [13intuition] 15Krinkle closed pull request #42: Remove reflinks (06master...06master) 02https://github.com/Krinkle/intuition/pull/42 [16:36:43] siebrand: good to see another challenge :) [16:37:14] sorry wrong channel [16:43:34] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL 20.00% of data above the critical threshold [0.0] [16:49:49] 10Tool-Labs: kmlexport perl script memory usage - https://phabricator.wikimedia.org/T99236#1292059 (10valhallasw) And again... ``` 20892 tools.kmlexport 20 0 322464 183984 3088 S 2.0 2.3 15:58.87 /usr/bin/perl /data/project/kmlexport/public_html//kmlexport.pl 9888 tools.kmlexport 20 0 272124... [16:59:42] [13intuition] 15Krinkle pushed 2 new commits to 06master: 02https://github.com/Krinkle/intuition/compare/181d11d30e62...a4263660080f [16:59:42] 13intuition/06master 14db44696 15Timo Tijhof: Remove unused *.i18n.php files... [16:59:43] 13intuition/06master 14a426366 15Timo Tijhof: Update Names.php from upstream MediaWiki master... [17:06:06] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Rafael Gattringer was created, changed by Rafael Gattringer link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Rafael_Gattringer edit summary: Created page with "{{Tools Access Request |Justification=Similar to the tool https://tools.wmflabs.org/?tool=stimmberechtigung I would like to give the user simple feedback, when he or she will..." [17:06:48] Is it possible to run queries against dbs on beta cluster? [17:11:05] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Rafael Gattringer was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=159597 edit summary: [17:13:35] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK Less than 1.00% above the threshold [0.0] [17:42:21] 6Labs: Officetools instance (running sugarCRM) is in suspended state and won't reboot). - https://phabricator.wikimedia.org/T99339#1292128 (10Andrew) 5Open>3Resolved a:3Andrew I started that instance, looks OK to me. [17:56:45] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Rafael Gattringer was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=159603 edit summary: [18:02:40] 6Labs, 10Beta-Cluster: Make it possible to run public queries against databases on beta cluster - https://phabricator.wikimedia.org/T99456#1292152 (10Glaisher) [18:03:04] 6Labs, 10Tool-Labs, 10wikitech.wikimedia.org: Promote @valhallasw to shellmanager on wikitech - https://phabricator.wikimedia.org/T99457#1292153 (10scfc) 3NEW [18:12:29] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Vivekiitkgp was created, changed by Vivekiitkgp link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Vivekiitkgp edit summary: Created page with "{{Tools Access Request |Justification=Developer - WIGI Project (Inspire grantee) |Completed=false |User Name=Vivekiitkgp }}" [18:17:49] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Vivekiitkgp was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=159605 edit summary: [18:18:12] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Vivekiitkgp was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=159607 edit summary: [21:56:21] [13intuition] 15Krinkle pushed 2 new commits to 06master: 02https://github.com/Krinkle/intuition/compare/ec9180d83d37...78418cc454d2 [21:56:21] 13intuition/06master 14baa5cb9 15Timo Tijhof: Add support for various deprecated language codes... [21:56:22] 13intuition/06master 1478418cc 15Timo Tijhof: tests: Clean up PHPUnit tests and improve caching... [22:06:44] [13intuition] 15Krinkle pushed 1 new commit to 06master: 02https://github.com/Krinkle/intuition/commit/fc1e599952b416d7586bff17a4d7c3a7325f5d15 [22:06:45] 13intuition/06master 14fc1e599 15Timo Tijhof: demo: Fix undefined message "apple-stats" in demo8... [22:15:18] [13intuition] 15Krinkle pushed 1 new commit to 06master: 02https://github.com/Krinkle/intuition/commit/700c7080e257287a04c0f600f4f1640cf159ee1d [22:15:19] 13intuition/06master 14700c708 15ChameleonWiki: Intuition: Generate valid html in getPromoBox()... [22:22:36] [13intuition] 15Krinkle pushed 1 new commit to 06master: 02https://github.com/Krinkle/intuition/commit/fe9cdd3a08e5ce25db45682ab246362c57fd1d18 [22:22:36] 13intuition/06master 14fe9cdd3 15Timo Tijhof: Intuition: Improve documentation of getPromoBox()... [23:24:11] does anyone miss the ascii unicorn that was in bastion?