Pixelastic

You can cut our wings but we will always remember what it was like to fly.

Posts tagged with "SSH"

Checking potential throttling from Orange

Starting a few weeks back, we started having painfully slow ssh connections at work. Our servers are hosted on the SoftLayer infrastructure, which are extended on all North American territory, but we are based in France.

We have fiber-optic connection here, almost every other connection is blazingly fast. Except connections to our own servers...

Call me paranoid, but the first idea we got was that we were being throttled by our provider. Orange is infamously known for degrading on purpose connection with some know hosts.

A colleague was browsing through his own secure tunnel, going through his personnal server for every outgoing communication.

And his connection was as fast as it should be.

That made us wonder. How a tunneled connection could be faster than a normal one ? The cost of crypting data should have slow the request down. It had the opposite effect, rendering the request in the expected time.

I created a garbage file using the following code and used scp to send it from one server to another, to test bandwith.

dd if=/dev/zero of=garbage.bin bs=100000000 count=1

Nothing interesting came out of it. I had a impressivly fast connection between our fiber connection and OVH (up to 50Mb/s) and slow connection to SoftLayer.

So, for the time being, I'm connecting to SoftLayer through OVH and even if it made me use one more tunnel, it is still far more fast than a direct connection.

Working on Wednesday #8 : Custom zsh scripts for housekeeping

I made a break of the Ruby/Rails learning this week. Instead I learned a bit more about the basic Linux command, and made a few scripts to help me in my day to day work (and fun) with my Linux environment.

I currently use two laptops. One is a tiny netbook while the other a Dell XPS. I mostly use the XPS for everything work related (like posting this), while the netbook is dedicated to casual use : browsing, playing, etc.

Last week, during my vacations, I only took the netbook with me, and while not strictly speaking working on it I still had to code a little bit. I installed ScummVM, CortixTH and managed to run Carmageddon 2 under wine. I also downloaded and tried a few old emulator games on my Dingoo.

As you may have guessed, I wrote a bunch of scripts during that week to help me in some of the more tedious aspects of this task.

Dingoo deployment

Downloading roms, copying them to the Dingoo, and then testing the games to make sure they are working was tedious. I wrote a little script that automatically mount the Dingoo when plugged to the computer, and copy my games to it, removing old version along the way while still keeping my saved games.

I had to parse dmesg (thanks to grep, tail and sed) as well as use rsync for the copy.

Sansa Clip cleaning

I also wrote a simple script to clean my Sansa Clip of all the useless files I had put on it. As I only have a text screen, I don't need the jpg covers, torrent txt files, auto-generated m3u files and other thumbs.db and .DS_Store useless files.

Here, find to the rescue with some zsh scripting, and here we go.

Shared config files

As I'm working on several computers (two laptops and the one at work at least), I found myself spending time reconfiguring stuff multiple times.

Here, thanks to Dropbox and some clever ln -s calls, I managed to replicate the same config on the 3 machines easily.

From my freelance time, I also have a pretty big /etc/hosts and ~/.ssh/config file. At work, I have another one for the work servers. But I sometimes had to connect to the work server from home.

So once again, thanks to Dropbox, some ln -s once more and cat I wrote two functions to regenerate those two files based on config files. That way, whenever I change one of those files, it is updated on the other machines too.

That was it. It took most of my day.

Being able to script those little repetitive tedious task is something I really appreciate on Linux. I am no longer dependent on the UI my OS gave me, and can now really understand how things work. It requires some time to learn, but this is not going to drastically change from one version to another like it does on each Windows new release.

Finding the perfect SSH host

I'm trying to find a company providing hosting capabilities. I have a strong set of pre-requisites.

Connected through SSH

I want the server to be accessible through SSH. I don't need http nor ftp acces on it.

I'll host git, mercurial and subversion repositories on it, and will push(/pull) from(/to) them using ssh.

So, I will need those softs installed, or at least enough power to install them (even locally).

Serving as an SSL tunnel

I also want to be able to create an SSH tunnel through this server. That way I could be securely connected from any wifi connection using SSL.

The server should have a pretty reasonable bandwidth. I'd also wish to avoid having it hosted in France. Peering agreement between providers in France are getting uglier and uglier.

Secondary accounts for holding websites

If the same company also provides more conventional hosting, that also could be an important part of my choice.

What I would like is something as flexible as allowing some websites to use Apache, and others Lighttpd. Even being able to configure it per subdomain.

Some sites would be running PHP, others Rails, even node.js. I want enough freedom to configure it and the associated inner config (lighttpd.conf, php.ini).

I also want to be able to install secondary parts like mongoDB or memcache and having a full access to their configuration.

Pricing

Of course, the cheaper the better, but I know that such an amount of freedom and features comes with its own price.

Now, let's review some of the contestants I've picked

Alwaysdata

They were the first I checked. I heard quite a good thing about their service and reliability. They really seem to know what they're doing.

I've tried their free 10Mo plan and am quite satisfied with it. Mercurial, git, subversion and quite a batch of others softs are already installed. They also have a pretty 10Mo/s awesome bandwith.

I haven't tried their hosting plans, but they also seems nice. All default (shared) hosting provides php, perl, ruby and python.

Their custom (managed dedicated server) provides almost anything you could want from lightty to memcache, but the price goes high with it to. 350€/month that is way too much I can afford for what I have in mind.

On the other hand, they not only provide hosting but the whole support package along with hotline and custom installation.

Dreamhost

I'm fond of Dreamhost. I've used them for years. They do provide a very nice shared hosting, with a lot of options to configure. They even allow cronjob for their admin panel.

And they are cheap, with a customer support very professional, I highly recommend them for your simple hosting needs.

I haven't tested their dedicated server version, so can't really talk about it.

However, the SSH capabilities of the shared hosting are quite small. They have git, svn and mercurial installed, but the version are quite outdated. Even the python version running mercurial was out of date.

You can however download and install them yourself on your account (which I did), but it is not as full featured as alwaysdata.

Their bandwith is also capped to 1Mo/s.

OVH

Haven't tested yet, but their VPS offer seems nice for as low as 60€/year.

I have tried the shared hosting offer of OVH before and am mitigated. On one hand I like some of their features like the automatic backup of your ftp files, on the other hand their administration panel is a total mess and accessing the server through SSH is not that simple.

I've heard they have a 10Mo/s bandwith. I'll try to get more information on this.

Gandi

Gandi is an amazing registrar (both technically and ethically). I've never had any issues with their service and can only praise their technical support.

Some years ago, they started doing hosting too. I tested it back then and was quite disapointed. This was horribly slow and crashed.

I never tried it again since then. I think they improved their service, and I just asked a test account. Their VPS offer is interesting, the server is pretty decent from what I can read and the price is still in my range (12€/month).

 

I'll continue this review later, with more information. If you, reader, have any suggestion on a good host filling my need, feel free to post it in the comments, I'll review it.

cakePHP deployment with Mercurial on Dreamhost

I now use Mercurial on my daily work flow and have set up some methods on my dev machine to ease the pain of installing mercurial and make it work on any new webserver.

Here are some snippets that automate all that stuff. You may have to change one thing or two to accomodate your own setup.

First, I create a custom .bashrc file that I will put on the webserver and create into the following method :

hgInstall() {
mkdir -p ~/.packages/src
cd ~/.packages/src
wget http://www.selenic.com/mercurial/release/mercurial-1.2.tar.gz
tar xvzf mercurial-1.2.tar.gz
cd mercurial-1.2
python setup.py install --home=~/.packages/

echo -e "[ui]\nusername = Pixelastic <tim@mailastic.com>" >> ~/.hgrc
echo -e "[extensions]\nhgext/hbisect=!\nhgext.imerge=! >> ~/.hgrc

. ~/.hgrc
cd ~/
hg version
}

Let me explain. I first create a directory to store the packages I will download (in this example I will only download one package, but as I don't like to have files all around my server, I just keep them in this place). I will then download Mercurial 1.2 in this new directory, unzip it and install it.

Next step is configuring the default user and correcting some bugs with Dreamhost trying to load non-existing extensions (hgext/hbisect and hgext.imerge). As I've made a change to .hgrc, I reload it and get back to the default directory while displaying hg version.

That's almost done, I also have to edit the .bash_profile and add the following lines

export PYTHONPATH=~/.packages/lib/python
export PATH=~/.packages/bin:$PATH

Ok, so this method will download, install on configure Hg on the Dreamhost server. That's all very well, but I had to manually setup the .bashrc, let's see if we cannot automate that as well.

Now, I'm editing my .zsh_aliases on my local machine (or your .bash_aliases if you're using bash) to add the following method

dreamhost() {
scp ~/Documents/Config/Dreamhost/.bashrc ~/Documents/Config/Dreamhost/.bash_profile $1:~/
ssh $1 '. ~/.bashrc'
scp ~/.ssh/id_rsa.pub ~/Documents/Config/Dreamhost/.ssh/xpsfixe.pub $1:~/
ssh $1 'addKeys'
scp ~/Documents/Config/Dreamhost/cakeClearCache.sh $1:~/
ssh $1 'chmod +x ~/cakeClearCache.sh'
ssh $1 'hgInstall'
ssh $1
}

Ok, so this one is a little more complex. You have to call this method with one paremeter, being the user@domain credentials to connect to your Dreamhost server. What it will do is upload (using ssh) files from your local machine to the server and then apply some commands on the machine using ssh.

First it will upload both the local version .bashrc and .bash_profile that are sitting on your dev machine and "reload" the .bashrc, allowing you to use the previously defined hgInstall directly in the shell

Then, it will upload your ssh key(s) to the server and add them to the list of allowed keys (more on that later, just skip the addKeys line for now.)

The next step is uploading (and giving the correct chmod) a special script that will clear cakePHP cache (more on that later too)

And the final step is calling the previously explained hgInstall method. So the only thing you have to do is put this method in your .zsh_aliases (and the corresponding keys, .bashrc and scripts in their corresponding places) then run dreamhost() and Hg will be installed on your server.

So now let me get back a little on the two details I skipped. The first is the key stuff. What I'm doing is uploading your ssh key(s) to the server and then calling addKeys. It will authorize those keys to connect using ssh without having to type login/pass on each request. Here is the addKeys code (you have to put it in your .bashrc file and modify the filename to your own)

addKeys() {
mkdir .ssh
cat id_rsa.pub >> .ssh/authorized_keys
cat xpsfixe.pub >> .ssh/authorized_keys
rm id_rsa.pub
rm xpsfixe.pub
chmod go-w ~
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
}

It will basically create the .ssh dir and authorized_keys file with your keys info. It will then delete the files and set the correct chmod.

And the second part was about that cache clearing thing. When you update your app using Hg, you do not want to update the cache files created by Cake as they contain filepath reference and are likely to be different between your test and prod environment and would surely broke your whole app. So, you set an ignore rule in the .hgignore about them like the following :

syntax:glob
app/tmp/cache/cake_*
app/tmp/cache/views/*.php
app/tmp/cache/models/cake_*
app/tmp/cache/persistent/cake_*

It does work fine almost all the time, but it sometimes lead to errors as the cache is not regenerated between each hg update. Sometimes you have to alter a model schema or the way a value is stored in cache and if you don't clear your cache, it can yield to unexpected results as the data will be wrongly parsed and used.

So what i did to avoid that was to create a script that will clear the cache for you. Here is the code (you have to be inside the project dir for this to work)

cd app/tmp/cache
rm -f cake_*
rm -f views/*\.php
rm -f models/cake_*
rm -f persistent/cake_*
cd ../../../

It will remove all the cache files generated by cake that could interfere after an update. You just have to wrap thoses lines in a method in your .bashrc (mine is called cakeClearCache) and execute it after each update or when you have caching issues.

Well, I think you guessed that I did not stop here. Manually applying the method after each update can be a little tedious. So I put the previous code in a file named cakeClearCache.sh (you can spot that I uploaded this file in the dreamhost() method earlie)r. I also added the following line to my /project/.hg/hgrc on my server (if you don't have this file, just create it, it's a project-based hg configuration file)

[hooks]
update = ~/cakeClearCache.sh

It means that everytime an hg update is done, the specified script is fired. That's really fine for us, it means that cache will be cleared on each update. Sounds good.

One last thing to do was creating the hgrc file automatically. That's why I created the following method (add it to the .bashrc file in the server). It is just a wrapper that will create the hgrc file after doing an hg init

hgInitStart() {
hg init
echo -e "[hooks]\nupdate = ~/cakeClearCache.sh" >> ./.hg/hgrc
}

So instead of doing hg init, just do hgInitStart. You can then start cloning your project here.

And one last thing, I also created a method that will set correct chmod to app/tmp and app/webroot/files

cakeCorrectChmod() {
chmod 777 ./app/tmp -R
chmod 777 ./app/webroot/files -R
}

And created a wrapper around it to call just after having cloned the project that will update it and set the correct chmods

hgInitEnd() {
hg update tip
cakeCorrectChmod
}

That's all. I bet anyone slightly more experienced in shell scripting could do better than that, but as I have struggled a little to get this right I thought I could share it.