We were unable to support messageboard search across all services in the same way we support article search because of the huge amount of posts in our system. We had to tackle the problem in a different way.

The messageboard search system has been architected to use a dedicated "Search" database. Resources dictate that this new database will initially live on the same server as the main messageboard database, but the architecture allows us to easily move the solution to one or more separate servers, as we bring search to more messageboards in the future.

One of the design goals was to make it easy to bring search to messageboards one by one, allowing us to provide effective and fast search without compromising the quality of existing services.

The diagram above shows the two databases. The messageboard database stores all the posts across all messageboards in a single table. The search database fetches the latest posts belonging to the messageboards that have been configured to support search. The posts for each messageboard are stored in dedicated tables in the Search database. We create a separate full text index on each table that allows us to efficiently gather search results on a per-messageboard basis, without having to filter the search results that come back from the engine. We make use of the ranking values the engine provides to order the results by relevance.

In order to provide the fastest and least resource-hungry searches, we've adopted a simple "AND" based search. We take the search term the user gives us, create a list of search words using the space character as the separator, remove any Stop words (i.e. common words that are no use in terms of search, such as "the", "and", etc.), and then ask the search engine to find all the posts that contain all the words in the term.

For example, if the term "Fish and chips" was passed in, we would first create a list of search words using the space character as the separator, yielding "Fish", "and" and "chips". We would then remove the Stop words, causing "and" to be removed, leaving "Fish" and "chips". We then ask the engine to find all posts that contain the words "Fish" and "chips". Incidentally, all searches are case insensitive.

The number of months' worth of posts that are searchable is also configurable, on a per messageboard basis. Again, this is to allow us to control the amount of server resource that messageboard search requires. The number of months of searchable content will be decided on how busy the board is, and the nature of the board. Some boards may be more interested in current posts, whereas other boards may have content that's more historical and therefore still valuable as time passes. Ultimately it is our intention to allow all posts on all messageboards to be searchable.

Mark Neves is lead database engineer, DNA team, Audience Publishing Services, Programmes and On-Demand, BBC Future Media & Technology