Empty Thoughts

Amazon.com just announced SimpleDB, a new offering in their Amazon Web Services line of products. Amazon SimpleDB is a web service for running queries on structured data in real time. It is ideally positioned to be used along side of EC2, their Elastic Cloud service, and S3, the Simple Storage service.

A lot of people have been asking for a structured data solution that is scalable and redundant. From a quick-look first glance it looks like SimpleDB is going to provide that answer. SimpleDB is not based on a relational database format but more of an open and flexible format that removes the need to maintain a schema, instead providing the flexibility to make modifications and adjustments as you go.

I’m not sure how all of this will work, but I’m certainly looking forward to getting my hands on an early beta.

I’ve written before how I use Amazon EC2 to handle my server needs for various startups I’m involved with. Amazingly, Amazon just announced new instance types to be used for EC2 to address larger instance needs.

Small Instance (default)

• 1.7 GB memory
• 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
• 160 GB instance storage (150 GB plus 10 GB root partition)
• 32-bit platform
• I/O Performance: Moderate
• Price: $0.10 per instance hour

Large Instance

• 7.5 GB memory
• 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
• 850 GB instance storage (2×420 GB plus 10 GB root partition)
• 64-bit platform
• I/O Performance: High
• Price: $0.40 per instance hour

Extra Large Instance

• 15 GB memory
• 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
• 1,690 GB instance storage (4×420 GB plus 10 GB root partition)
• 64-bit platform
• I/O Performance: High
• Price: $0.80 per instance hour

One of the challenges in the past has been trying to figure out how to tie together a lot of separate smaller instances when you need large centralized storage or or a more beefy centralized application server. These new large and extra-large instance types make that problem go away and give you tremendous growth opportunity.

I’m continually amazed at the things going on over at AWS.

Django on Gentoo (DoG) Linux image. This is built on the base Gentoo image that I posted about previously . I added a whole bunch of goodness to run Django. Included are a whole set of startup scripts that build up the system and initialize the application.

AMI ID: ami-29947140
AMI Manifest: eminent-ami/django/10/image.manifest.xml

About this AMI

This image contains a Gentoo Linux 2007.0 installation with Django application support. The following applications / libraries are installed and/or configured:

• Gentoo Linux
  • dhcpcd,
  • logrotate,
  • all packages are recent and updated,
  • AMI tools have been installed and are fully working even for volume imaging,
  • EC2 meta-data is fetched from 169.254.169.254 into /var/spool/ec2/meta-data,
  • EC2 meta-data and user-data include files in /var/spool/ec2/meta-data and /var/spool/ec2/user-data respectively,
  • Disabled password authentication in /etc/ssh/sshd_config
  • Modifications to startup according to Amazon documentation
  • the portage tree has been bind-mounted to /mnt/usr/portage to preserve space on the root file system,
  • the /tmp directory has been bind-mounted to /mnt/tmp to preserve space,
  • the /var/lib/postgresql directory has been bind-mounted to /mnt/var/lib/postgresql,
  • the root home directory contains a dev-copy directory that is needed to build the volume images,
  • all packages have been recompiled to correctly respect the Xen environment.

• Django
  • Apache 2.2.4 installed (not in startup) and configured with Python support,
  • Python 2.5.1,
  • mod_python,
  • Postgresql 8.2.4 (not in startup),
  • psycopg2
  • Subversion 1.4.3 (no repositories are setup)
  • Django trunk version checked out into /home/django/django and installed as normal into the Python site-packages directory (django-admin.py copied to /usr/local/bin/),
  • Support packages: libjpgeg, zlib, freetype2, zip, and unzip,
  • Python packages: PIL 1.1.6, FeedParser 4.1, Markdown 1.6, Unipath 0.1,

• This image contains the following daemons / services:
  • local
  • net.eth0 / net.eth1
  • netmount
  • dhcpcd
  • sshd
  • syslog-ng
  • vixie-cron
  • apache 2 (disabled by default)
  • postgresql (disabled by default)

Sample Application

The instance can accept a set of user data parameters for setting up an application. I’ve provided a default application to indicate the process for creating your own application setup scripts. If you run the instance with the following parameters it will startup a complete running default application based on the Django Vote tutorial.

ami-29947140 -k gsg-keypair -d "app=mysite-setup=http://s3.amazonaws.com/eminent-ami/mysite-setup.tgz"[/code]Once the server is up and running you should be able to to enter the public hostname (starts with ec2) into your browser and view the application. The Admin section of the application is in the location /admin. Credentials are:

Username: admin
Password: password

The default application will be installed into the /home/django/mysite directory. The setup scripts for the application will get installed into /opt/mysite-setup. The Admin media directory and the mysite media directory are both symbolically linked to the /var/www/mysite/htdocs directory.

Postgresql Configuration

Postgresql is installed but there is no database setup and the service is not started by default. The database directory /var/lib/postgresql is bind mounted to /mnt/var/lib/postgresql. The configuration files have been configured according to the Gentoo Postgresql How-To (http://gentoo-wiki.com/HOWTO_Configure_Postgresql) document with the included logging changes. Configuration files are symbolically linked in from /etc/postgresql for those that like that sort of thing. A sample of creating and restoring the database can be found in the mysite-setup scripts for the Sample Application. Write Ahead Logging (WAL) is not implemented, but would be necessary implementation detail for most sites.

Creating an Application Setup Package

The mysite-setup package (http://s3.amazonaws.com/eminent-ami/mysite-setup.tgz) is the base requirements for your own application setup package. At a minimum you are required to have a bin directory with a file in it called setup.sh. Within the setup.sh script you are free to put any setup instructions you need or pass those tasks off to other scripts written in the language of your choice.

Currently the configuration of this image requires that the application setup package is publicly available and downloadable via curl (see the application-setup.sh script). I plan to add security implementation details in the future.

Thanks

Grateful thanks to Geert Bevin for his initial Gentoo image.

Special thanks to RightScale for ideas and guidance provided by their startup scripts

After months of waiting patiently, I was finally informed that my elastic cloud account was available. If you’re not familiar with all of the cool things Amazon Web Services is doing, I recommend you spend some time getting acquainted with their offerings. I’ve been using S3, their storage solution, for some time now, and I’m definitely stoked about playing around with EC2. This on the heals of Jesse Newland’s announcement of Capazone, a Capistrano task library for managing EC2 servers. Like I needed one more thing to distract me from all the projects I should be working on.