Datasports on Software Development

Articles and updates from Datasports about the craft of software

Running Effective Proof of Concept Projects

leave a comment »

1. Introduction

Lately I’ve had a few people wanting to discuss their proof of concept and/or vendor & tech evaluation projects with me, specifically around evaluating StreamBase for their needs on upcoming projects and products. Seems like as good a topic as any for an article, so here we go. My first use of StreamBase was when I was driving a PoC project for my then-employer, so I feel like I have some insight to offer on the subject, based on that and on my years of experience since.

There are 4 broad categories to consider when planning and executing any project like this: effort and duration of the PoC, project definition and scope, choosing the right resources, and applying the correct evaluation criteria. This article will offer some guidelines and suggestions for those 4 categories, specifically as they apply to evaluating StreamBase for your organization or project.

NOTE: This article covers PoC projects for the core StreamBase CEP platform, writing EventFlow logic and using provided Adapters. It does not cover anything specific to evaluating the LiveView product.

2. Effort and Duration

The first question is how much time and energy to invest in your PoC project. I would contend that this depends to some degree on the scope of the project for which you intend to use it, but I recommend running a project that involves at least 2 people for 2 weeks, and not more than 4 people for 4 weeks. My recommendation, then, is for a project that is 2-4 weeks in duration, and 4-16 weeks of effort.

I suggest having at least 2 people involved, because with a single person there is too much opportunity for the impressions to be perceived as biased or skewed by one person’s preferences. Of course, that perception could also be correct – if a single resource is irrationally fond of, or displeased with StreamBase the organization won’t get the careful analysis that they require. Having at least 2 people on the project makes it more likely that the final results are based on thoughtful evaluation, rather than individual whim. I advise against having more than 4 people involved because with a higher headcount, the project management overhead becomes a larger percentage of the total PoC effort, which is not germane to the evaluation of StreamBase specifically.

As for duration, I feel that less than 2 weeks is just not enough time to really get a feel for the platform. Figure 3-5 days to start really getting a feel for how to use it, and then 5-7 days to crank something out that can be evaluated. On the other side of the coin, I think that past 4 weeks you would just be getting into diminishing returns territory. For sure there’s a lot to learn about the platform, more than can really be learned in a month or two, but it’s extremely unlikely that anything will come up past that 4 week mark that significantly changes the evaluation of the platform.

3. Project Definition and Scope

It is important is to make sure that you are defining a project that’s actually well-suited to StreamBase, otherwise you’ll end up with skewed results. I’ve written a piece that discusses what is and what isn’t a good fit, along with some other stuff about productivity in StreamBase here:
StreamBase and Productivity

The short story about choosing a suitable project is:

Systems that map well to StreamBase’s event processing model are real-time systems which will consume traffic in the form of well-defined messages or events from disparate systems, and which respond to requests or certain conditions by sending out events or messages to other systems.

Just about all real-time financial services systems are an excellent fit for StreamBase (especially anything related to electronic trading), and it is in this domain where StreamBase has found its most receptive clients to date. However, there are many other problem domains that fit this general high-level description, such as sensing and data acquisition, healthcare informatics, biometric monitoring, electronic messaging, and innumerable others. It remains to be seen if customers from these other problem domains will adopt StreamBase and push the platform to grow in new directions.

Systems that are not a good fit for StreamBase include anything that does complex recursive batch processing on large static datasets, intensive graphics rendering, processing of data whose format is not known at design time and must be inferred from the data itself, and systems where performance is unimportant.

I would suggest that the whole article is worthwhile reading for someone who’s planning to do a careful evaluation of StreamBase. Once you have selected an appropriate type of project to use for your PoC, you can proceed to the next steps.

3.1. Training

StreamBase offers a range of options for training developers new to the platform. They can work with you to present material that’s most relevant to the focus of your PoC, and offer in-depth training for up to a week or more. I suggest that with the limited timeframe of most PoC projects, plus the fact that the quick ramp-up is a big part of StreamBase’s value proposition, you should arrange for a total of 1-2 days of training, done in person or online.

In-person training with a trainer from StreamBase professional services is preferable, but may not be possible depending on the constraints of your project. If you can manage it, I think it’s a good idea to include some additional people who may not be participating in the actual PoC to take part in the training, at least for the introductory discussion of the core platform. This can help foster interest, and provide context for the results of the PoC.

3.2. Functional Requirements

I suggest writing a fairly ambitious list of specific requirements, and prioritizing them carefully, using just 3 levels of priority:

  • Priority 1 is something that’s required for the PoC to be considered successful. Except under exceptional circumstances, StreamBase will be considered to have failed the evaluation unless everything on this list is completed.
  • Priority 2 is something that definitely adds to the value of the project, either in terms of exploring relevant functionality or design patterns, or in making a compelling demo.
  • Priority 3 would be “sizzlers”, something that’s not necessary for evaluating the SB platform, but that would make PoC demos more exciting/compelling.

Do all #1 items first, then based on that experience, reexamine the #2 and #3 items, and proceed to do all #2’s, followed by any #3’s that you have time for, or that have some strategic value. Because timelines are generally tight for these types of projects, and because the people doing the work will generally be unfamiliar with the strengths of the platform and appropriate design patterns, it is a good idea to review progress and priorities at the start or end of each day. You may find that something you first thought was critical is trivial as you learn more about the platform, or that it’s more work than expected and that your time would be better spent on other items.

3.3. Connectivity

Integration with external systems is often high on the wishlist for PoC projects, and it certainly makes for compelling demos. I suggest that you should integrate with systems for which there is a StreamBase adapter available (e.g. for market data and/or FIX order entry), but do not invest PoC effort in writing custom adapters for your proprietary internal systems. If you want to include custom adapter implementation as part of your PoC, then have a look at the StreamBase Component Exchange (SBX) for something that is similar to what you want to do, and use that as a starting point. To get a sense of what’s involved in writing an adapter, you can also have a look at my tutorials for the Datasports DirectoryWatcher Adapter and the Datasports WebServiceListener Adapter.

NOTE: Do not turn your evaluation of the StreamBase platform into an exercise in writing custom Java code, or you will miss the point. Keep the focus on developing EventFlow code, and using the provided adapters.

There are a variety of options for simulating external systems to which you can’t connect during your PoC, whatever the reason. For one PoC project I worked on, we used the opportunity to evaluate VeriFIX by Greenline alongside StreamBase, and we used it to submit orders, as well as to simulate multiple execution venues. This approach was effective as it let us include another important tool in our evaluation without adding significant effort. Other approaches include using canned data & replaying it using mechanisms such as the CSV Input Adapter, or StreamBase’s built-in stream simulation tools.

It is also worth noting that many brokerages, execution venues, and providers of market data offer simulation tools and test environments that may be of use in your PoC. Call them and ask what they can do to help with the integration side of your evaluation.

3.4. Delivery and Deployment

StreamBase applications can run on a variety of desktop and server platforms, with features provided for monitoring and debugging remote systems. Even so, for a PoC project, I recommend running your components on desktop development machines if possible. Deployment and server management with StreamBase is not especially difficult, nor is it appreciably different from deploying and running a straight Java application to a server, so I suggest leaving that part of the work out of your PoC, and staying focused on doing as much development with the core StreamBase platform as possible in the time allotted to your project.

NOTE: Developing, building, and running under the StreamBase Studio IDE (based on Eclipse) is the fastest and easiest way to work, which makes it the best-suited to a PoC project in my opinion.

4. Resources

Care should be taken to ensure that the people taking part in the PoC represent the right mix of skills in order to make the project as successful as possible – not just in terms of delivering the most complete product, but in gleaning the most relevant insights about the platform.

To facilitate the work of the dev team working on the PoC, I recommend having someone from StreamBase professional services on site for the first week of the project, and on call thereafter. Having someone on site isn’t really about using them 100% of the time that they are present, it’s about ensuring that they’re there, ready to respond as soon as they’re needed in order to help keep the work moving forward. Also, it never hurts to have an expert on site to overhear questions and discussions between members of the PoC team and offer insight.

Plans for having someone on call should include some kind of solution such as Skype or other online meeting systems that make it easy for developers with questions to quickly share their canvas, test data, and results with their assigned StreamBase resource.

As for internal resources: I believe that you’re best off to use a mix of people who are senior and junior, hopefully with relevant domain experience. For example, if you’re going to integrate with a FIX order entry system, then at least one person on the team should be familiar with FIX. Likewise for the business logic: if you’re making a basic order router, then have at least one person who has worked on order routing or execution management in the past. To properly evaluate the platform, you want to involve people who won’t be spending their time ramping up on domain knowledge.

NOTE: Try to choose the members of your PoC team and support them such that they will be able to concentrate as much of their time as possible on building EventFlow code. This will put them in the best position to make an effective evaluation.

5. Evaluation

Your evaluation criteria should be stated as clearly as possible at the start of the project. A big part of this is in how you define your Priority 1 requirements, but you can specify other criteria as well, and these can be a mix of quantitative and qualitative items. If your concern is the ability to rapidly adapt/adjust business logic in response to changing circumstances, were you able to modularize your logic to isolate business logic from infrastructure? If your concern is scaling out via a variety of partitioning schemes, were you able to define your system in a way that facilitates different deployment configurations? If you know you need to operate across unreliable connections, then how does the system behave when remote (or simulated) external systems disconnect and reconnect?

How did the developers respond to the visual programming model? What were the surprises, positive and negative? Did system performance meet, exceed, or fall short of expectations? Was StreamBase professional services helpful and responsive?

NOTE: Do your best to lay out these questions at the outset of your evaluation, based on some defined goals for the project so that when the PoC development effort has completed you can re-focus on the high level goals in a rational manner.

6. Conclusion

In this article we have reviewed some general principles and practices for running an effective PoC project to evaluate the StreamBase platform for your organization’s needs. The basic principles are to set clear objectives, prepare by front-loading the training activity, arrange to have the appropriate resources involved, and keep them focused on the core parts of the system: writing, running, and debugging EventFlow code in the StreamBase Studio IDE and configuring input and output adapters.

If you have found this article interesting or helpful, please take some time to post your comments, questions, or suggestions on this blog.

Thanks,
Phil Martin
Datasports Inc.

Advertisements

Written by datasports

Jun 14, 2012 at 9:14 AM

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: