Pixelastic

I'm trying to find a company providing hosting capabilities. I have a strong set of pre-requisites.

Connected through SSH

I want the server to be accessible through SSH. I don't need http nor ftp acces on it.

I'll host git, mercurial and subversion repositories on it, and will push(/pull) from(/to) them using ssh.

So, I will need those softs installed, or at least enough power to install them (even locally).

Serving as an SSL tunnel

I also want to be able to create an SSH tunnel through this server. That way I could be securely connected from any wifi connection using SSL.

The server should have a pretty reasonable bandwidth. I'd also wish to avoid having it hosted in France. Peering agreement between providers in France are getting uglier and uglier.

Secondary accounts for holding websites

If the same company also provides more conventional hosting, that also could be an important part of my choice.

What I would like is something as flexible as allowing some websites to use Apache, and others Lighttpd. Even being able to configure it per subdomain.

Some sites would be running PHP, others Rails, even node.js. I want enough freedom to configure it and the associated inner config (lighttpd.conf, php.ini).

I also want to be able to install secondary parts like mongoDB or memcache and having a full access to their configuration.

Pricing

Of course, the cheaper the better, but I know that such an amount of freedom and features comes with its own price.

Now, let's review some of the contestants I've picked

Alwaysdata

They were the first I checked. I heard quite a good thing about their service and reliability. They really seem to know what they're doing.

I've tried their free 10Mo plan and am quite satisfied with it. Mercurial, git, subversion and quite a batch of others softs are already installed. They also have a pretty 10Mo/s awesome bandwith.

I haven't tried their hosting plans, but they also seems nice. All default (shared) hosting provides php, perl, ruby and python.

Their custom (managed dedicated server) provides almost anything you could want from lightty to memcache, but the price goes high with it to. 350€/month that is way too much I can afford for what I have in mind.

On the other hand, they not only provide hosting but the whole support package along with hotline and custom installation.

Dreamhost

I'm fond of Dreamhost. I've used them for years. They do provide a very nice shared hosting, with a lot of options to configure. They even allow cronjob for their admin panel.

And they are cheap, with a customer support very professional, I highly recommend them for your simple hosting needs.

I haven't tested their dedicated server version, so can't really talk about it.

However, the SSH capabilities of the shared hosting are quite small. They have git, svn and mercurial installed, but the version are quite outdated. Even the python version running mercurial was out of date.

You can however download and install them yourself on your account (which I did), but it is not as full featured as alwaysdata.

Their bandwith is also capped to 1Mo/s.

OVH

Haven't tested yet, but their VPS offer seems nice for as low as 60€/year.

I have tried the shared hosting offer of OVH before and am mitigated. On one hand I like some of their features like the automatic backup of your ftp files, on the other hand their administration panel is a total mess and accessing the server through SSH is not that simple.

I've heard they have a 10Mo/s bandwith. I'll try to get more information on this.

Gandi

Gandi is an amazing registrar (both technically and ethically). I've never had any issues with their service and can only praise their technical support.

Some years ago, they started doing hosting too. I tested it back then and was quite disapointed. This was horribly slow and crashed.

I never tried it again since then. I think they improved their service, and I just asked a test account. Their VPS offer is interesting, the server is pretty decent from what I can read and the price is still in my range (12€/month).

I'll continue this review later, with more information. If you, reader, have any suggestion on a good host filling my need, feel free to post it in the comments, I'll review it.

One can protect the browsing of a special directory with a simple set of login/password by using appropriate .htaccess/.htpasswd files.

The classic way

Just create an .htaccess in the directory you want to protect with the following lines :

AuthName "Restricted Access"
AuthType Basic
AuthUserFile /full/path/to/your/.htpasswd
<Limit GET POST PUT>
Require valid-user
</Limit>

And to create the .htpasswd file, run the following command :

htpasswd -c /full/path/to/your/.htpasswd username

The -c modifier will create the file, omit it if you only want to add a new user. Also change the path to your .htpasswd file (moving it out of the webdir could be a good idea) and change username to any login you want.

You'll then be prompted to enter the password (twice) and your file will be generated.

cakePHP and Dreamhost fixed

I had an issue when protecting a folder in my app/webroot/ folder on Dreamhost. I'm not sure it is completly cake related nor Dreamhost related but the two together made it quite hard to debug.

Anyway, it appears that when issuing an HTTP Auth, Dreamhost redirect to a file named /failed_auth.html (this is the file you're supposed to see when your Auth fails, obviously).

But as I didn't have such a file in my app, everytime I tried to access my protected dir, I got my custom 404 error page.

To finally fix that, all I had to do was to create a real failed_auth.html page, or in my case, create a Route that redirect failed_auth.html to a custom failed auth page.

I guess just dropping a failed_auth.html file in app/webroot/ could have done the trick too.

Dreamhost as a wonderful wiki on how to set cron jobs and cakePHP has a cookbook page on how to do it too, but getting it just right was a little frustrating.

I finally managed to run my shell task as a cronjob, and here is how I did it.

The magic command

First, here is the command I wrote in my Dreamhost panel :

/home/username/domain.com/cake/console/cake -app /home/username/domain.com/app/ shell_name

You have to set the full path to the cake console because it obviously is not in the pre-saved paths on a default Dreamhost install.

You also have to specify the app directory otherwise cake will search your shell in the wrong directories. This is even more important if your shell is located in a plugin directory.

But, this won't work as-is, you'll have to do several other stuff, mostly explained by Miles Johnson, but I'll recap them

Setting the file as executable

You have to set the cake/console/cake file as executable. It is kind of obvious, but you have to do it anyway.

Set the TERM variable

Update you cake/console/cake to replace the first lines with :

LIB=${0/%cake/}
APP='pwd'
TERM=linux
export TERM

If you don't, you'll have notice errors in the resulting log. No big deal, but it is cleaner that way.

Forcing using php5

While in command line, the php command refer to the php4 version. If you need to use php5 (and I guess you should), you would have to manually reference the binary file. Hence, you have to update your cake/console/cake file one more time and change the exec line to :

exec /usr/local/php5/bin/php -q ${LIB}cake.php -working "${APP}" "$@"

And it's ok

Your cronjob should not work effortlessly. It took me some long hours to track all this down (along with other issues on my local dev machine that make debugging even more cloudy).

I have previously blogged on how I migrate mails from one IMAP server to another. Today I had to migrate some mails again.

My client's staff wasn't happy about the GMail webmail we gave them. The lack of a true "Draft" feature (one tha could be used and reused) had them drop a lot of productivity, so we decided to get back to Squirel Mail.

I started by manually recreating all the email adresses on the back-end on the Dreamhost server. Once at least one address is created, Dreamhost automatically creates the matching A and MX records.

I had to update my zone file on my registrar to update the previously MX and A records that were set to point to Google and updated to point to Dreamhost. I have to use an intermediate zone file on my registrar because I'm dealing with a limitation of the .fr TLD here.

Once the zone were updated and changes repercuted accross the network, I connected to the new Squirel mail. My inbox was empty.

So I ran imapsync to copy all mails from the google hosted service to my new server.

The syntax to be used is a little different than the one I mentioned on my previous post, so here it is :

imapsync --host1 imap.gmail.com --ssl1 --authmech1 LOGIN --user1 foo@domain.fr --passfile1 /path/to/pass1 --host2 208.97.*.* --user2 foo@domain.fr --passfile2 /path/to/pass2 --useheader="X-GMail-Received" --useheader 'Message-Id' --noauthmd5

After that, I reloaded my Squirel Mail inbox, and it was populated. I just had to do that for every account.

Recently, when connecting to my MySQL databases on Dreamhost through Navicat, I was greeted with a "1577 - Cannot proceed because system tables used by Event Scheduler were found damaged at server start" error message.

It didn't stop me from accessing the tables actually, so I didn't bother. But today it prevent me from copying tables from one server to another, so I decided to have a deeper look into it.

It seems that it has to do with a mysql upgrade issue. I contacted Dreamhost about that and they told me that they just upgraded their servers to 5.1, and that they needed to be restarted. Which they do for me.

So if you ever run into the same problem, just contact Dreamhost support and they'll fix it for you.

One of my clients wanted to change its hosting provider because of almost constant downtime. I suggested that we move the whole site and the associated email addresses to Dreamhost.

He is heavily using its webmail, with a dozen of different accounts, all with custom address book and drafts to reuse.

I had already moved a previous site of the same client to a GMail powered webmail and he wasn't so satisfied of it because of the lack of a "draft" feature that would allow him to reuse the same message templates.

Its current webmail is powered by Imp (Horde) and the default Dreamhost webmail is Squirel Mail.

So now I'm faced with a challenge. Which webmail should I choose and how will I initiate the migration ?

Contestant 1 : Horde

The easiest way would have been to reuse Horde. Horde has a nifty export/import function. It means that I could have exported the settings on the current host and imported them back in the new host.

Unfortunatly, Horde is not pre-installed on Dreamhost. It means I would have to install it and I don't really want to do it. This is not my area of expertise and if something goes wrong it will take me ages to fix.

Additionnaly, the default php5 install on Dreamhost lacks the IMAP extension needed and SSL connection requires an additional monthly fee.

That would be the better solution for the client, but I don't feel capable of doing it.

Contestant 2 : Squirel Mail

Squirel Mail is the default webmail on Dreamhost. It means that they do handle all the heavy stuff of configuring it and making sure it does not crash.

They assure me that I can have unlimited hosting of my mails, the only limits are a maximum of 100 SMTP sending per hour and no attached filed > 7Mo.

Those are almost no issues at all, we can handle that. I just have to check if the 100 sending per hour is for each mail address or for the entire account.

The only problem will be that I can't export/import the horde feature into Squirel.

What I could do however is synchronize the two IMAP servers, effectively copying all the mail from the current one to the new. Unfortunatly, there is no easy way to do that.

The Dreamhost support even told me that it was impossible. I found a linux command named imapsynch however that maybe could help, but I'll need to know the name of the dreamhost imap server and needed credentials to connect to it. I'll sort this out with them.

Contestant 3 : GMail

I've already set mails up with GMail a couple of times in the past so this won't be too hard. Could take some times, but nothing difficult about that.

I may be able to copy the current mails to the GMail server, I've seen some articles on the subject floating on the network, but I'll definitly won't be able to copy drafts.

Conclusion

I'm still in contact with the Dreamhost support to choose the better option. I don't think I'll install Horde, this will be too much work. I guess I'll go with the second option of migrating mails to Squirel Mail.

Whatever solution I'll pick, I'll be sure to post the whole details here for others to follow.

Update

I finally won't have to migrate mails, the client makes regular POP backup of its mails so he will just do one more right before the transfer and will start over with a brand new mail boxes. Concerning the drafts, he will manually copy-paste them from one webmail to the other and he told me that he wasn't using the address book...

So it appears that I'll just have to change the DNS to point to the new webmail and create a temporary CNAME to still access the old webmail on the old server until the migration is totally over.

Re-update

In fact, it appears that I won't be able to access both webmail at the same time. I'll have DNS issues, and the registrar panel that I have is very basic, I won't be able to fine tuned it correctly.

It seems that I'll have to migrate mails from one server to the other.

Trying to push some new code to a Hg repository on my Dreamhost account, I had the following error message :

remote: *** failed to import extension hgext.imerge: No module named imerge

I used to have the same kind of issues with an other extensions, hgext/hbisect a while ago and I fixed it by forcing Hg to not use this extension.

Here's how :

Edit your .hgrc file and under the [extensions] category, add hgext.imerge=!, like this

[extensions]
hgext/hbisect = !
hgext.imerge=!

This should stop the warning.

Yesterday I moved this domain from one server to another on my Dreamhost account (actually, I moved the whole domain from one Dreamhost account to another, but that's the same).

The DNS saved at the registrar where the same (ns1.dreamhost.com, ns2.dreamhost.com and ns3.dreamhost.com). The transfer went seamlessly and I was able to access my website on the new server in an invisible way.

But, strangely, some hours laters I started to experience slow down when accessing the website and finaly all my requests timed out, be it either from Firefox or a direct ping.

I tried with my windows xp and my Ubuntu 8.04. I even change the DNS on xp to use opendns, run an ipconfig /flushdns, rebooted. On Ubuntu I've done a /etc/init.d/dns-clean start. I even rebooted my adsl box, thinking there was something wrong with the DNS cache on my side.

I even contacted both Dreamhost and Gandi with a complete traceroute to know if they could help me, as I thought having done everything that I could think of to resolve the issue. They assure me that their configuration was fine and the problem wasn't on their side.

When asking friends, or testing with http://downforeveryoneorjustme.com/ it occured that the problem really was on my side.

Everyone were true. The problem lied somewhere in between, in the DNS cache hold by my ISP that wasn't correctly up to date. I don't really understand how that could happen, I thought rebooting my Freebox would clear its cache, and specifying dns addresses of opendns would bypass the dns resolve of my ISP but it seems not to be the case.

Anyway, after waiting for some more hours, my website was again reachable through ssh and ftp.

So, if you're sure your dns configuration is correct both on your register and on your hosting service, make sure to clear every DNS cache you can have on your own side. That include windows inner dns cache, those of your browser, those of your router, and those of your specified preferred DNS resolving server.

If all that failed, it should be caching issue with your ISP, and there is nothing you can do about that, except waiting.

Hosting a website on a .fr is a little harder than a simple .com. The Afnic (registry for the .fr domains) makes a check whenever you change the DNS of your domain to make sure the new DNS can accept the domain.

They run a tool named ZoneCheck to do that. This tool will request the new DNS server to test if they can accept the .fr. The problem is that they make a request on the port 53, which is not open on Dreamhost and thus the test fail and the Afnic refuse to let you change your DNS.

The alternative is to not change you DNS, and keep the default one of your registrar, but manually edit the zones and copy the zones defined in your dreamhost panel (In Domains > Your domain > DNS, at the bottom of the page) into your zones.

The most important part is modifying the default A entry to the IP of your dreamhost server. You should do the same for the www A entry to.
Create a mysql A entry and a ftp A entry if you want to use both ftp.domain.com and mysql.domain.com.

I also changed the MX record to the google one, as well as creating a CNAME webmail.

I now use Mercurial on my daily work flow and have set up some methods on my dev machine to ease the pain of installing mercurial and make it work on any new webserver.

Here are some snippets that automate all that stuff. You may have to change one thing or two to accomodate your own setup.

First, I create a custom .bashrc file that I will put on the webserver and create into the following method :

hgInstall() {
 mkdir -p ~/.packages/src
 cd ~/.packages/src
 wget http://www.selenic.com/mercurial/release/mercurial-1.2.tar.gz
 tar xvzf mercurial-1.2.tar.gz
 cd mercurial-1.2
 python setup.py install --home=~/.packages/

 echo -e "[ui]\nusername = Pixelastic <tim@mailastic.com>" >> ~/.hgrc
 echo -e "[extensions]\nhgext/hbisect=!\nhgext.imerge=! >> ~/.hgrc

 . ~/.hgrc
 cd ~/
 hg version
}

Let me explain. I first create a directory to store the packages I will download (in this example I will only download one package, but as I don't like to have files all around my server, I just keep them in this place). I will then download Mercurial 1.2 in this new directory, unzip it and install it.

Next step is configuring the default user and correcting some bugs with Dreamhost trying to load non-existing extensions (hgext/hbisect and hgext.imerge). As I've made a change to .hgrc, I reload it and get back to the default directory while displaying hg version.

That's almost done, I also have to edit the .bash_profile and add the following lines

export PYTHONPATH=~/.packages/lib/python
export PATH=~/.packages/bin:$PATH

Ok, so this method will download, install on configure Hg on the Dreamhost server. That's all very well, but I had to manually setup the .bashrc, let's see if we cannot automate that as well.

Now, I'm editing my .zsh_aliases on my local machine (or your .bash_aliases if you're using bash) to add the following method

dreamhost() {
 scp ~/Documents/Config/Dreamhost/.bashrc ~/Documents/Config/Dreamhost/.bash_profile $1:~/
 ssh $1 '. ~/.bashrc'
 scp ~/.ssh/id_rsa.pub ~/Documents/Config/Dreamhost/.ssh/xpsfixe.pub $1:~/
 ssh $1 'addKeys'
 scp ~/Documents/Config/Dreamhost/cakeClearCache.sh $1:~/
 ssh $1 'chmod +x ~/cakeClearCache.sh'
 ssh $1 'hgInstall'
 ssh $1
}

Ok, so this one is a little more complex. You have to call this method with one paremeter, being the user@domain credentials to connect to your Dreamhost server. What it will do is upload (using ssh) files from your local machine to the server and then apply some commands on the machine using ssh.

First it will upload both the local version .bashrc and .bash_profile that are sitting on your dev machine and "reload" the .bashrc, allowing you to use the previously defined hgInstall directly in the shell

Then, it will upload your ssh key(s) to the server and add them to the list of allowed keys (more on that later, just skip the addKeys line for now.)

The next step is uploading (and giving the correct chmod) a special script that will clear cakePHP cache (more on that later too)

And the final step is calling the previously explained hgInstall method. So the only thing you have to do is put this method in your .zsh_aliases (and the corresponding keys, .bashrc and scripts in their corresponding places) then run dreamhost() and Hg will be installed on your server.

So now let me get back a little on the two details I skipped. The first is the key stuff. What I'm doing is uploading your ssh key(s) to the server and then calling addKeys. It will authorize those keys to connect using ssh without having to type login/pass on each request. Here is the addKeys code (you have to put it in your .bashrc file and modify the filename to your own)

addKeys() {
 mkdir .ssh
 cat id_rsa.pub >> .ssh/authorized_keys
 cat xpsfixe.pub >> .ssh/authorized_keys
 rm id_rsa.pub
 rm xpsfixe.pub
 chmod go-w ~
 chmod 700 ~/.ssh
 chmod 600 ~/.ssh/authorized_keys
}

It will basically create the .ssh dir and authorized_keys file with your keys info. It will then delete the files and set the correct chmod.

And the second part was about that cache clearing thing. When you update your app using Hg, you do not want to update the cache files created by Cake as they contain filepath reference and are likely to be different between your test and prod environment and would surely broke your whole app. So, you set an ignore rule in the .hgignore about them like the following :

syntax:glob
app/tmp/cache/cake_*
app/tmp/cache/views/*.php
app/tmp/cache/models/cake_*
app/tmp/cache/persistent/cake_*

It does work fine almost all the time, but it sometimes lead to errors as the cache is not regenerated between each hg update. Sometimes you have to alter a model schema or the way a value is stored in cache and if you don't clear your cache, it can yield to unexpected results as the data will be wrongly parsed and used.

So what i did to avoid that was to create a script that will clear the cache for you. Here is the code (you have to be inside the project dir for this to work)

cd app/tmp/cache
rm -f cake_*
rm -f views/*\.php
rm -f models/cake_*
rm -f persistent/cake_*
cd ../../../

It will remove all the cache files generated by cake that could interfere after an update. You just have to wrap thoses lines in a method in your .bashrc (mine is called cakeClearCache) and execute it after each update or when you have caching issues.

Well, I think you guessed that I did not stop here. Manually applying the method after each update can be a little tedious. So I put the previous code in a file named cakeClearCache.sh (you can spot that I uploaded this file in the dreamhost() method earlie)r. I also added the following line to my /project/.hg/hgrc on my server (if you don't have this file, just create it, it's a project-based hg configuration file)

[hooks]
update = ~/cakeClearCache.sh

It means that everytime an hg update is done, the specified script is fired. That's really fine for us, it means that cache will be cleared on each update. Sounds good.

One last thing to do was creating the hgrc file automatically. That's why I created the following method (add it to the .bashrc file in the server). It is just a wrapper that will create the hgrc file after doing an hg init

hgInitStart() {
 hg init
 echo -e "[hooks]\nupdate = ~/cakeClearCache.sh" >> ./.hg/hgrc
}

So instead of doing hg init, just do hgInitStart. You can then start cloning your project here.

And one last thing, I also created a method that will set correct chmod to app/tmp and app/webroot/files

cakeCorrectChmod() {
 chmod 777 ./app/tmp -R
 chmod 777 ./app/webroot/files -R
}

And created a wrapper around it to call just after having cloned the project that will update it and set the correct chmods

hgInitEnd() {
 hg update tip
 cakeCorrectChmod
}

That's all. I bet anyone slightly more experienced in shell scripting could do better than that, but as I have struggled a little to get this right I thought I could share it.