shell_sink

Moved code to github.

2011-07-14T23:51:00.000-07:00

Yea this project is mothballed, but thought I would move the code over to github just in case.

Public Shellsink instance shutdown

2010-12-16T23:20:00.000-08:00

I've turned off history.shellsink.com. It makes sense since all the source code is in launchpad, and since hosting your own private shellsink on appengine is free. Anyone still using shellsink can migrate to their own private shellsink server. And now I can stop paying for the domain name and storing people's shell commands on my appengine account. Thanks to everyone who contributed to the project.

Introducing shell_sink

2010-12-09T15:53:00.000-08:00

What is it?

Shell Sink is a web accessible version of your bash history. Commands issued in bash are sent to shell_sink by a simple open source client application. Once a command is in shell_sink you can search, annotate and tag it.

Why use it?

If you've ever wasted time trying to remember a particular command line incantation, shell_sink is for you. It stores commands and makes them easily accessible with searching and tagging. If you work on multiple computers shell_sink can aggregate your history across all machines. System administrators can even use it to aggregate commands issued across an entire organization making them fully searchable. Never lose a command again.

Who should use it?

shell_sink is for anyone with a unix system. Linux, BSD, OSX, Solaris, Playskool. It should be noted that shell_sink is currently under active development.

Getting started:

Login to shell_sink with your google account.
Copy your unique id from the Preferences page.
Install the client application.

Questions, Issues:

The shell_sink google group is the place for questions and discussion. The bug tracker is where to put your feature requests and issues pertaining to the shell_sink website. The client is open source software. Source code can be found here.

Installing the shell_sink client application

2010-11-28T07:11:00.000-08:00

Installing the sink client should be a snap. There are only 2 steps:

Install the shellsink-client application
Configure a couple environment variables

The client is a python application that you can call from the command line to asynchronously upload a command and relevant tags to the shellsink server. When properly configured it will run behind the scenes every time a command is issued. It can also be easily disabled or enabled for times you don't want to log commands. Here are detailed instructions for installation:

Debian Based Distributions
If you have a Debian based distribution you can install shellsink-client by adding the shellsink repository to your apt sources. Go to this page for instructions and follow the instructions under the link that says "Technical details about this PPA." Once you have added the shellsink repository you can update your package lists and install the client with

sudo apt-get update
sudo apt-get install shellsink

Finally, you must append this file to your ~/.bashrc to enable the client. Be sure to replace the default tags with your own meaningful tags and most importantly set the default id to whatever you see here.

Other Types of Unix
For other flavors of unix you can download the shellsink-client executable here and put it in your path. Once the client is installed you must set a few environment variables. Append the contents of this file to your .bashrc or .bash_profile. If you use zsh see this how-to. Be sure to change the id variable to be the one that was assigned to you by the shellsink server on the preferences page. Also change the tags. I always start with the name of the host machine as my base tag so I know which computer logged which command.

Finally, just close your terminal and reopen it. Type a command and go see if it has been logged to the shell_sink server.

ShellSink adds inline tagging

2009-09-28T20:23:00.000-07:00

Release 0.2.2 adds a new feature. You can add tags to a command at the same time as you execute it. This is achieved by using inline comments. Here is an example:

cat /proc/cpuinfo #sysadmin:diagnostic

That command will show up in your shellsink history with the tags sysadmin and diagnostic. This nifty idea was given to me by Jesse Hallett at my shellsink presentation at the open source bridge conference in portland this summer. Thanks Jesse.

Filtering What Goes Into Your History

2009-05-02T16:23:00.001-07:00

When I tell people about ShellSink, many of them say, "hey, that's a great idea!" Then the little sysadmin perched on their shoulder crosses her arms and shakes her head. "Tsk, tsk. Storing your shell history on a remote server is not secure!", she says. This is absolutely true. Not only is your bash history passed via an insecure Http connection, but then it is stored unencrypted on Google's servers where a rogue Google employee, someone who hacked Google, or a rogue ShellSink admin could access them. Don't use shellsink on a system that needs to be ultimately secure!

However, Shellsink is safe to use. If you use Google for email, calendar, documents, etc. then you already trust them with information that is probably more sensitive than your shell history. And if you don't trust me you can download the ShellSink source code and spin up your own server using Google's AppEngine. In any case you should still take some basic precautions. Most importantly, never use inline passwords!! ShellSink will send anything your bash history captures.

Here are several ways to safeguard yourself against sending sensitive information to ShellSink. One technique is to add the following to your .bashrc:

HISTCONTROL=ignorespace;

Now any command you precede with a space will NOT be captured by the history, and thus will not be sent to the ShellSink server. Another technique is to add the following to your .bashrc:

HISTIGNORE="netstat*:sqlplus*"

In this example any command that starts with netstat or sqlplus will automatically be ignored by the shell history and ShellSink. Both these techniques give you different granularity control over what gets sent to ShellSink. The first technique works on a per command basis and the second covers all commands.

Finally, there is the nuclear option:

shellsink-client -d

This completely disables ShellSink for all terminals. I don't really have much use for this one, but when you start to do something that you are ultra paranoid about, this is one way to be sure nothing is leaving your computer via ShellSink. To re-enable ShellSink simply do this:

shellsink-client -e

ShellSink at the Open Source Bridge Conference

2009-05-02T15:55:00.001-07:00

I'm going to be in Portland for the Open Source Bridge Conference (OSB) to talk about ShellSink. The abstract I submitted is overly ambitious as I promised to cover shellsink usage, implementation as well as distribution via Launchpad. I've decided to stick to usage and implementation of shellsink as I am giving a presentation entirely dedicated to distributing software via launchpad at OSCON this year. Of course, OSB has the totally awesome 24hour hacker lounge as well as an entire day dedicated to uncoference antics and open space shennanigans, so if anyone feels shortchanged I'd be happy to spend some time talking about launchpad (or whatever) in any of those situations. Please drop a line if you are going to be at OSCON or OSB this year.

Shellsink Nearing Free Quota

2009-02-26T21:29:00.001-08:00

Shellsink has been a pet project of my for the last few months, and it has been fun to put something new out there just to see how people react. My main goal of course was to build something useful. What I didn't anticipate was that I would have to begin paying for hosting so soon. Unfortunately Google has rolled out the non free portion of app engine, and they have announced resulting changes to the service. I had hoped that ad revenue on the shellsink site would pay for whatever app engine hosting costs were incurred, but since 99% of use is at the command line, actually very very few ad impressions are viewed on the site.

The short of this is that very soon shellsink will hit the data storage quota. 1GB of commands, tags, and annotations will have been logged and indexed on the shellsink site. I an trying to get Google to lift the quota for my application. But I have little hope unless one of the Googlers using the service decides to intervene. I expect that late next week shellsink.com will hit the quota and stop accepting commands. BUMMER!

The good news is that all data you have logged there will remain accessable, but if you are like me, that isn't much consolation. So the other good news is that all the source code for the server is going to be in the shellsink Launchpad bazaar repo, so you can keep using shellsink by deploying to your own app engine account. So, it looks like without a deux ex-machina that shellsink.com will have to morph into a non-centralized system.

UPDATE: There is a thread in the google group about this.

Using shellsink with ZSH is easy

2009-02-08T06:50:00.000-08:00

You don't have to use bash to use shellsink. Shellsink will work with any shell that supports the following features:

Concept of a command that always executes before your command is run
Concept of a history file
Concept of appending to the history file each time you run a command

Z-Shell is another great shell and when configured correctly it supports all these things. Here is a set of instructions to make shellsink work for zsh.

Grab the sample zshrc
Concatenate that to ~/.zshrc
make a symlink from .bash_history to .zhistory like this:
```
ln -s ~/.zhistory ~/.bash_history
```
Restart your shell

Easy, right? Here is a list of my References:

http://en.gentoo-wiki.com/wiki/Zsh#Shell_History
man zshmisc
man zsh

ShellSink Server Upgrade

2009-01-19T21:05:00.000-08:00

On the same day that our government gets an upgrade, ShellSink gets an upgrade. Specifically, a new version of the server is live! The changes are all performance related and it enables me to bring the client behavior into line with the way google has designed app engine to scale. That is, to make lots of very simple requests instead of fewer complex requests.

Unfortunately, this version is not backwards compatible with older versions of the client. You must upgrade your client! Why would I make such a drastic change? Simple, nobody is using shell_sink but me! That means I'm sure 100% of my users will upgrade. It doesn't explain why I need a blog entry to tell myself about it.

How to use tags in the sink.

2009-01-07T23:14:00.000-08:00

Tags can be useful in many different ways. I thought I would share some of the ways I use tags.

Using tags to say what computer has logged the command: I set the SHELL_SINK_TAGS environment variable in my bash_profile to indicate what machine logged a particular command like this:
```
SHELL_SINK_TAGS=myhost
```
This is particularly useful because I let several machines log their commands to the same shell-sink account, and if I didn't do that I wouldn't be able to remember which machine logged what.
Using tags to group commands: Often times when I am working on a particular project I want to indicate that the commands I am issueing are related to that project. This makes it easy to remember the context of commands in my history and helps me find things later. Let's say I have a project called "gloop". When I set down to work on gloop I'll set my tags environment variable to indicate that as follows:
```
export SHELL_SINK_TAGS=$SHELL_SINK_TAGS:gloop
```
Now any commands issue in the terminal I am working in will get tagged with gloop. When I want to stop tagging with the gloop tag I simply issue the following command
```
source ~/.bash_profile
```
. If your SHELL_SINK_TAGS variable is defined in a different file than mine you will need to source the appropriate file for your environment. On ubuntu that file is ~/.bashrc
Using tags to control your RSS feed: If you go to the preferences page of your shellsink account you will see that you can turn on an rss feed. You can also filter your rss feed by any tag, so if you only want people to be able to see certain commands in the feed you can choose the appropriate tag from the droplist. As soon as you select something in the droplist only commands with that tag will appear in your RSS feed.
Using tags to pull: The shellsink-client application normally just pushes your commands out to the server, but it can also pull commands from the server and put them into your history file. If you issue
```
shellsink-client -pt gloop
```
. The most recent 20 commands tagged with gloop will be pulled into your bash history file. This is especially useful for making all the commands for a particular task immediately available in your history file.

These are just a few of the ways you can use tags in shellsink. Some things to remember while using tags. Currently you can only add a maximum of four tags at a time. When adding multiple tags from the client make sure you use a colon delimited list of tags. When adding multiple tags from the web application use a comma delimited list of tags. Enjoy, and please leave your own recipes for working with tags here in the comments.

Photo Credit: sarah parrott

Why search terms are restricted.

2008-12-27T14:25:00.001-08:00

Currently you can only search for two keywords at a time. Shellsink uses Google App Engine (GAE) as the underlying technology. The keyword search implementation provided by GAE is currently quite limited. There is: no substring matching, limited number of keywords and only fields of 500 characters or less are indexable. I know. It is a bummer. Unfortunately, search is a complex problem and I don't have the brains to solve it. I'm going to wait for google to solve it. I predict later versions of the App Engine APIs will improve the search functionality and I look forward to removing these limitations just as soon as Google does.