Author: mike

Do you need to monitor your redis data store, but rather than see everything under the sun only want to see the messages published in the pubsub?

redis-cli monitor | grep -E ' "PUBLISH" '

Piece of cake, and so helpful for tracing through to find the source of issues in a complex system.

With some server shuffling happening recently this blog appeared to be on the chopping block. Didn’t take much poking around to remember how much history there is here and feel like it was worth finding a new server home, and hosting platform. 1012 blog posts from 2003 to 2013! That is a body of work! I suppose of debatable value.

Despite it being 4+ years since I last posted, and 14+ years since I started writing on MovableType (which is now only available commercially), there was still a very usable article on migrating to WordPress (as Stephen Rosenberg says “the software that swallowed MovableType’s user base whole”).

2003 was a long time ago, an eternity in internet years. Before smart phones and tablets. Before Hybrid cars. Before mainstream adoption of blogs, when blogging was something a small number of geeks did. I’m really happy that all 1012 posts migrated with everything intact, pictures, links. Even more, the instructions included details for maintaining URL structure so links to posts and search engine indexing would work seamlessly.

Next post scheduled for 2021 🙂

The Ruby on Rails app I’m building from the ground up needs to be able to handle *lots* of data. From the get-go I’ve known I’d need to put some kind of data partitioning to make this thing scale, but just finally tackled it this week.
Rails doesn’t come with a built-in ability to connect to multiple databases, but there are lots of folks who have tackled it. Ryan Tomayko’s post Rails and Scaling with Multiple Databases (written way back in 2007) is a good indicator that there are folks doing serious business with rails and multiple databases, and has some good links to other sources.
I was looking at writing something myself, but in digging around came across Octopus a few times and realized it does exactly what we need, data sharding with the ability to programmatically select databases in controllers and models, as well as good migration handling.
For the application we’ll have a master database which centralizes account, user and session information with a pointer to the appropriate data shard. Once a user request has been authenticated, the actual application data gets pulled off of one of the shards they’ve been assigned to. The master database has a simple schema for user, account, and session information, the schema on the shards is significantly more complex, but the same on each shard.
Octopus documentation is there, but not extensive. Through some trial and error I found that the following shard.yml configuration file was what I needed (the Octopus documentation doesn’t spell out how to use shard groups in a rails environment, but it makes sense once you get it working). The master shard for the environment is defined in database.yml.

octopus:
environments:
- development
- test
- production
development:
data_shards:
data_1:
adapter: postgresql
database: dev_data_1
username: something
password: cryptic
template: template0
data_2:
adapter: postgresql
database: dev_data_2
username: something
password: cryptic
template: template0
...

It’s obviously paramount that the models know where to connect for their data. When a user authenticates I invoke ActiveRecord::Base.establish_connection() to connect to the shard so it’s the default data source everywhere in the application for that request. Requests that need to interact with the master database are handled in the controller:

around_filter :select_shard
def select_shard(&block)
Octopus.using(:master, &block)
end

The other important piece of this is migrations. Octopus works well for our needs here. When a migration is written it either needs to update a single master database or be applied to all data shards. This is where shard groups come in. To apply a migration to all shards we specify the group with using_group() method, and the migration then runs for each shard in the specified group.

class CreateInventoryEvents < ActiveRecord::Migration
using_group(:data_shards)
...
end

Pretty slick, happy with Octopus from a functionality perspective. Not sure about scalability and the overhead to move requests between database connections, will dig more into that next.

Listening to Chad Dickerson (CTO at Etsy) speaking about optimizing for developer happiness at RailsConf 2011.
Etsy releases software 25 times a day with 75 engineers. They emphasize constant progress with a radical decentralization. It shouldn’t work, but they are built on trust. There is one button that folks can press (after some fanfare) that pushes code up to production
They’ve had developers push code on their first day, they’ve had board members deploy code.
Chad contrasts industrial revolution assembly line times with how Etsy works. Etsy highly prioritizes letting the developers be human, having lunches together, having offices that are full of creative, fun stuff. Dogs have deployed code by guiding a paw to pressing the button.
Releases are done with a self-managed push queue, developers put their names in a queue kept track of in an IRC topic. When the first person in line pushes and is clear he/she notifies the next in line.
Remember, your team is your community. Help people finish things and see their progress.
Optimize your teams for the happiness of people.

Fantastic closing keynote by Glenn Vanderburg at RailsConf 2011 about software engineering.
Great ideas about software engineering and how it compares to classical engineering disciplines (civil, structural, industrial, mechanical, electrical, traffic, etc). Compares the definition and process of structural engineering to software engineering.
Is software engineering or craftsmanship? In software, the documentation that is required by other disciplines is actually the artifact that gets built in software engineering. Classical engineers diagram, plan, document, model and then take those and build the artifact. Software engineers diagram, plan, document, and model as a part of building the actual artifact. Also interesting, software is changing the way classical engineering works, a lot of software being used in the planning process to understand how the actual artifact
We are designers and builders. Software has to be the solution and describe the solution.

Programs must be written for people to read and only incindentally for machines to execute.
Harold Abelson and Gerald Jay Sussman

We are engineers and craftsmen.

In the RailsConf 2011 RubyGems presentation this afternoon, Nick Quaranto spent some time talking about versioning software, showing the versioning of TeX as kind of crazy where they use PI, adding a new digit of PI on the end of the software version number for each release.
Nick suggests that Semantic Versioning is the best methodology. Read more at http://semver.org/
The idea is:
1.2.3 (major.minor.patch)
major versions – for major releases where there may be some backward incompatible changes
minor versions – for minor changes, no API or other changes that make it incompatible with previous versions
patch version – for fixes, improving stability