Pixelastic

As I'm about to move for a couple of long months without an internet connection and a lot of travel hours, I thought of downloading the TEDTalk podcasts.

Being a linux user for about a year now, I'm taking the habit of using the linux tools to do the boring and repetitive work. Downloading more than a hundred files seems like a good candidate for a script.

The complete list of audio podcasts is available here. I simply selected all text and copied it into a file.txt.

Then, it was simply a matter of extracting only the urls, and feeding them to wget

cat file.txt | tr ' ' '\n' | tr '\t' '\n' | grep http > list.txt
wget --no-clobber -i list.txt

This took a while to download as there are more than 400 files, but that way I'll have some interesting talks to listen to during my trip.

I made a break of the Ruby/Rails learning this week. Instead I learned a bit more about the basic Linux command, and made a few scripts to help me in my day to day work (and fun) with my Linux environment.

I currently use two laptops. One is a tiny netbook while the other a Dell XPS. I mostly use the XPS for everything work related (like posting this), while the netbook is dedicated to casual use : browsing, playing, etc.

Last week, during my vacations, I only took the netbook with me, and while not strictly speaking working on it I still had to code a little bit. I installed ScummVM, CortixTH and managed to run Carmageddon 2 under wine. I also downloaded and tried a few old emulator games on my Dingoo.

As you may have guessed, I wrote a bunch of scripts during that week to help me in some of the more tedious aspects of this task.

Dingoo deployment

Downloading roms, copying them to the Dingoo, and then testing the games to make sure they are working was tedious. I wrote a little script that automatically mount the Dingoo when plugged to the computer, and copy my games to it, removing old version along the way while still keeping my saved games.

I had to parse dmesg (thanks to grep, tail and sed) as well as use rsync for the copy.

Sansa Clip cleaning

I also wrote a simple script to clean my Sansa Clip of all the useless files I had put on it. As I only have a text screen, I don't need the jpg covers, torrent txt files, auto-generated m3u files and other thumbs.db and .DS_Store useless files.

Here, find to the rescue with some zsh scripting, and here we go.

Shared config files

As I'm working on several computers (two laptops and the one at work at least), I found myself spending time reconfiguring stuff multiple times.

Here, thanks to Dropbox and some clever ln -s calls, I managed to replicate the same config on the 3 machines easily.

From my freelance time, I also have a pretty big /etc/hosts and ~/.ssh/config file. At work, I have another one for the work servers. But I sometimes had to connect to the work server from home.

So once again, thanks to Dropbox, some ln -s once more and cat I wrote two functions to regenerate those two files based on config files. That way, whenever I change one of those files, it is updated on the other machines too.

That was it. It took most of my day.

Being able to script those little repetitive tedious task is something I really appreciate on Linux. I am no longer dependent on the UI my OS gave me, and can now really understand how things work. It requires some time to learn, but this is not going to drastically change from one version to another like it does on each Windows new release.

This morning I started reading a lot of online blog post, framework description and other webdev material online I had in my bookmarks. After reading them, I saved them on my pinboard account to find them easily later.

Lot of good JS and CSS framework/libraries in there, I hope to be able to test them soon.

I also check my mailbox and respond to mails from my normal job. I shouldn't. I'm on a day off, I have to force myself to completly cut out those external calls.

Now, let's the work begin

Anyway, I finally finished the vimtutor. I still have a some difficulties to remember the mapping of the basic hjkl movement keys (I always want use k to move down instead of j). However, I feel confident enough in it to try writing my new code in it.

So, I headed to the Rails for Zombies tutorial. I thought it was free but the "win $5 worth of online teaching" / "next course costs $5" make me doubt it a little. Well, we'll see.

After registering to it, I wanted to save my login and password to KeePass as I always do.

Unfortunatly, KeePass on my Ubuntu machine works by using mono, and the keyboard interaction weren't that good (caret is slightly misplaced, inputs lags before getting displayed, text selection is weird). So I thought "Hey, let's see if there is a new version". Big mistake.

It appears that there was indeed a new version, a real one, with apt-get and stuff while mine was some hackish install I manually made a few month ago.

But wait, this means manually adding a ppa directory. No big deal, I trust the author. What ? Still doesn't work ? Hmm, I see. I need mono 2.6, and I only have 2.4.

What ? 2.4 is the latest available version on Ubuntu 10.10 ? You mean I must upgrade to 11.04 ? Hmm. Why not. I'll have to do it eventually, so let's go.

So I ran all my updates, to have the most up to date system. Unfortunatly, language-pack-gnome-fr doesn't want to upgrade so the OS upgrade isn't displayed.

Wait, what am I doing ?

And that's when I remember that my initial wish was to learn Ruby on Rails and that I was now on the road to upgrading my whole operating system. That is not exactly speaking procrastination, but it makes me do so much other things that my initial goal that I really looks like it.

I finally managed to purge my packets, upgrade to 10.10 and then to 11.04. But this took me 4 hours, and now it's 11PM and I guess I won't start this rails tutorial today, after all.

I'll finally have to migrate mails from one IMAP server to another because of a change in the hosting provider of one of my clients.

Let me first do a quick review of my current config :

I have an IMAP server on a not-so-reliable host. I need to move all the mailboxes to a brand new Dreamhost server.

The current host use horde/imp as its webmail, but Dreamhost is running under Squirel Mail.

How to do that ?

When I first contacted Dreamhost to ask them the better way to copy the content of my old IMAP server to their new one, they told me that IMAP to IMAP transfer was unfortunatly not possible.

After some digging I found a little linux tool called imapsync that was developped to synchronize two IMAP servers. As I only wanted to copy content from one server to an empty one, I guess it should work.

And it does.

Installing the tool

First of all, you really need to have a Linux machine. I'm sorry for all Windows and Mac users out there, but this solution will only work with Linux.

Obviously, you'll first have to download and install it.

I'm a newbie linux user, and don't use it at its full potential. It means that I rely a lot on the UI, I don't know most of the more basics commands.

So, I don't know the correct procedure to install something, I always use the Synaptic Packager Manager UI. I just have to type imapsync, check its checkbox and start the install.

I guess you, more experienced linux user, would know how to do that anyway.

Running the command

Once it's installed, you'll just have to run one command. But before typing it, you have make sure that you have all the required informations.

You'll need the server name, login and pass of both the source and destination account.

In my case I had previously reset all the password to newly generated one on the source server, then created the same email (with same password) on the destination server.

For the source server name, you shouldn't have any problem finding it. It usually is something like mail.domain.com.

But for the destination server, that's a little more tricky. As I haven't yet changed DNS, I have a Dreamhost server but no domain name pointing to it. So I can't use mail.domain.com there.

I had to go my Dreamhost DNS panel (clicking on DNS under the name of my domain in Manage domain) and check for the IP address associated with the mail A entry.

Once you have all this, create a file (name it pass) which contain the password of your account (or pass1 and pass2 if source and destination password aren't the same). This is done so you won't type in clear text the password of your accounts, or they could be found in the history files.

Once it's done, just run the following command

imapsync --host1 mail.domain.com --user1 contact@domain.com --passfile1 /path/to/pass1 --host2 208.97.XXX.XXX --user2 contact@domain.com --passfile2 /path/to/pass2 --noauthmd5

It is not an instant process, actually in can be quite long. One of the mailbox I had to move had more than 20.000 mails in the sent folder and about 40.000 in the received one. It took me almost 18h to complete.

I don't usually change MX resolving, but when I do, it always almost is using my registrar or host admin panel.

When I have to manually do it, (because the website is hosted on a dedicated server where I have to manually edit almost everything), I always waste a lot of time understanding where to find the configuration files.

Today I had to change the MX record of an existing website to use those of Google. The server hosting the website was also defined as nameserver, so I had to update the /var/named/domain.tld.hosts file (the file may not be in the same directory in your own server).

I remove the line relative to the MX records and added those instead

domain.tld.    IN    MX    10 ASPMX.L.GOOGLE.COM.
domain.tld.    IN    MX    20 ALT1.ASPMX.L.GOOGLE.COM.
domain.tld.    IN    MX    20 ALT2.ASPMX.L.GOOGLE.COM.
domain.tld.    IN    MX    30 ASPMX2.L.GOOGLE.COM.
domain.tld.    IN    MX    30 ASPMX3.L.GOOGLE.COM.
domain.tld.    IN    MX    30 ASPMX4.L.GOOGLE.COM.
domain.tld.    IN    MX    30 ASPMX5.L.GOOGLE.COM.

It worked well.

I also wanted to use a simple address for the webmail, like mail.domain.tld.

So I added the following line

mail.domain.tld    IN    CNAME    ghs.google.com.

I also updated the serial at the start of the file and rebooted my server. I know there must be a less savage way that rebooting the server but I still haven't find it.

P.S : I used this google / DNS Stuff page to help me test my records all along the way

Ok, so remember when I was writing about that client of mine who had trouble with its SquirelMail implementation. I thought it was a problem of SquirelMail having trouble dealing with large directories.

It was not exactly that, it more was a question of the IMAP server having issues with disk space. I made some backups of the mails, cleaned it up a little then everything went fine.

But suddenly today, the same SquirrelMail started acting funny, blocking some login attempt telling me that the IMAP server stopped responding, or displaying message list without subject nor owner.

Tired all of this, I decided (my client) to move its whole mail system to GMail, using the Google system, while keeping my clients domain name. To finish the registration process at Gmail, I had to put a googlehostedservice.html file online on the domain to prove that I am the owner and...

452 Transfer aborted.  No space left on device

Here is what the FTP server answered. What ? No more space ? But, but, but, but I just cleaned it all, deleting a whole 1Go log file. Wtf ?

I connect to the server and got the list of the biggest files and directories by doing a :

find  /var -type f -ls | sort -k 7 -r -n | head -10

I found that the maillog was 1.1Go... I deleted it and hurry the Gmail activation process. Problem solved.