The grid

Popular amongst the late 90’s geeks as broadband and unlimited RTC plans reached the home, self hosting has fallen into disuse as companies started to provide better, managed services, all for free. It was nearly impossible for open source projects like Imp or Squirrelmail to compete against Gmail, and the billion fueled services had an enterprise class credibility some bearded guys in their garage could not afford.

Things have changed after Snowden’s leaks started to raise privacy concerns and companies ending in the deadpool with all their data made people realize they had no other protection than good will when the service is free. A massive « get your data back » campaign started, helped by the fact that setting up a Web server has never been easier. By an incredibly ironic step back in the past of shared time computing, running your own server on the Internet is just 1 click and a few cents an hour away.

10 years ago, my CTO told me something that almost ruined my career.

System administrator is the dumbest job on earth. All you need to get things working is read the manual and Google for tutorials to cut / paste.

He forgot to add « until real people start using your platform ».

Indeed it’s very easy. Create an Amazon Web Services Account, download a the latest version of Zimbra, CozyCloud or Owncloud, or use ready to use Docker images and you’re ready to start host your data. Until something gets wrong.

Sooner tonight, David Meyer brought the topic back on Twitter.

I want to set up a personal server (mail, calendar, storage, personal webpage) in the cloud. Recommendations for a host?

That’s a good question, because there’s no easy answer.

If you value your data and want to host them in your own cloud, pick up a managed service or hire a professional system administrator to install and run it. Otherwise, leave them to Google, they have people who know what they’re doing.

The past 10 years, we’ve been moving most of our digital selves from Desktop software to browser or apps based portals to online services. We started to vote online, pay our taxes online, or do our supermarket online. Our lives are now so unmaterial that losing your data or have them compromised has became more annoying than losing your passport during a trip in North Korea.

If you (or anyone reading this) want to host their cloud without hiring a professional sysadmin or going into a managed service, here are the 3 main issues you’ll have to address.

Backups

If there’s one thing no one tell you about when you start self hosting your data, it’s backups.

I’ve lost all my data only once.

The first time was the 28 June 2001. It was around 7:00 AM. I had spent the night coding on a project and Slackware 8.0 was just released. I did not want to wait to upgrade, and when I started the install, I told the installer to delete /home instead of /. Back then, hard drives were still expensive and I had no backups of my data.

Backups are not about leaving Time Machine transfer your data to your home Time Capsule while you’re away. It’s about saving the right stuff, encrypt them before you send them to a 3rd party service, ensure it doesn’t fail silently and it restores well.

If you want to sleep quietly, you should test your backups at least once a month to ensure they restore. If you don’t, it’s like you’re not doing backups at all.

Security

The Internet is not a nice place to hang out. Thousand people are looking for weak machines they can compromise either to steal and sell the data when it’s worth it (ask French Domino’s Pizza what they think about it), or use them to mine Bitcoins or join a botnet to run denial of services.

I’ve been compromised at least 3 times. I say « at least » because you’re never sure when it happens unless it gets so obvious you can’t ignore it and get your machine shutdown by your hosting company. The first time was in February 2002. Back then, I was hosting a bunch of machines in my 14 square meters room in Paris. I was sleeping and got awaken by the noise of my mail server hard drive scratching as if someone was running a find /, which is probably the case. I jumped out of my bed and pulled the plug before I went back to bed.

Security is not just subscribing to your operating system and software mailing list, looking for advisories to update things. It starts when you start reading the famous tutorials and change a few things to ensure services run with different, unprivileged users instead of root, bind servers on a UNIX socket instead of 0.0.0.0, or run in containers / different virtual machines / jails, you name it.

It’s also checking your logs daily, looking for some unusual activity, and learn to recognize when your system is under attack or compromised. It’s setting up your mail server well enough so recipients won’t consider you as spam. Or simply setup a good backup policy.

It’s finally knowing about the legal stuff around self hosting and being compromised. You probably don’t want the police to ring your bell at 6 AM because the cloud you’re hosting your family pictures on has been compromised and used for a large operation against some US bank, or to host kiddie porn.

Availability

When you’re relying on an application provider, starting with Gmail for example, you’re confident they have enough servers, data centers and network providers to ensure redundancy so your data are still available when there’s a flood in Virginia. If they’re really good, they’ll also replicate your data in various places so they won’t lose it if the data center burns, which may happen someday.

Without going that far, there’s alway a risk when start self hosting your data that it becomes temporarily or permanently unavailable.

Imagine that worst case, not so uncommon scenario. You need to upgrade some service on your self hosted cloud. You follow the documentation, but things don’t go well and the service becomes unavailable. Indeed, you’re hosting some business critical things, like your professional address book or all your documents, and you won’t be able to reach them for maybe 1 hour, 1 day, you don’t really know how much time it will last. You won’t be able to get your email either, and you didn’t setup a MX backup so you’re completely unreachable for an undefined amount of time.

Scary isn’t it?

This is the real topic when people start to talk about self hosting. Once you’ve got your data back and the excitement is over, there’s a non negligible amount of daily – often boring – work you’ll have to do instead of watching the latest Lolcats video on Youtube.

Perry the Platypus wants you to subscribe now! Even if you don't visit my site on a regular basis, you can get the latest posts delivered to you for free via Email: