Quote Service

Jul 23, 2010 at 9:15 PM

I am interested in using laharsub to stream stock quotes to a silverlight client.  I downloaded the code and made some quick changes to the in memory backend provided and I am now able to stream quotes for one stock.  I am not sure about the best way to proceed from here.  What I did so far is add a timer to the backend that is instantiated when a topic is created.  The timer calls a method to get stock quotes over tcp from a dll and then calls begin publish for the topic ID.  I am interested to know the best way to go about publishing multiple stock quotes.  Should I pass all of the symbols to the subscribe method?  I don't need the users to publish anything but I am not sure if I should create a new topic for each symbol or have one topic that serves multiple quotes for multiple symbols?

Jul 23, 2010 at 9:34 PM

Laharsub allows you to model stock quote subscribtions in a number of ways. You can have a topic associated with a set of stock symbols (a "basket"), or one topic per symbol. One important consideration is that the granularity of a topic you choose will apply to both subscriptions and publications. The approach that offers most flexibility is one topic per symbol. Laharsub allows every client to subscribe to multiple topics (symbols) and handles this case very well by multiplexing multiple subscriptions on a single HTTP long poll request at the protocol level.

In terms of publishing new stock quotes to the system, I would suggest considering decoupling this mechanism from the Laharsub server implementation. The process that generates new quote values should act as any other client of Laharsub and invoke the appropriate REST API to publish a message to a topic. This allows for full encapsulation of server functionality in the Laharsub server which will compose better with any future enhacement and changes in the server side functionality (e.g. related to scale-out).

Please note that a Silverlight and .NET client should be available by end of this weekend.

Jul 26, 2010 at 4:02 AM

The new release of Laharsub available at http://laharsub.codeplex.com/releases/view/48730 adds support for Silverlight clients. This release contains:

  1. .NET 4.0 client with unit tests at 65% code coverage
  2. Silverlight 4 client with unit tests
  3. jQuery client
  4. .NET 4.0 WCF HTTP pub/sub server based on an HTTP long polling protocol, hostable as a Windows Service or a console application.

Thanks,
Tomasz

Aug 4, 2010 at 4:14 PM
Thanks Tomasz. I built a separate service to publish quotes to Laharsub and decided on the one topic per symbol model. I also implemented the Silverlight and .NET clients and they are working great so far. I am currently building a solution to translate the topic id into a symbol and I'm not sure of the best way to do this. I have considered changing Laharsub and the clients to accept a string instead of an integer as a topic Id since all symbols we use are unique. I have also considered an Http Module to translate the topic Id to symbol by storing the mapping in a database. I may have to change the Silverlight client for the latter approach but it will make Laharsub version upgrades easier. What do you think is the best approach? Are there any plans to expand the topic Id to include strings?
Aug 5, 2010 at 2:33 AM

I have intentionally designed Laharsub to define a topic only as a unique numeric topic identifier, without attempting to associate any topic metadata or application specific semantics with that identifier. This approach introduces the minimal set of concepts necessary for Laharsub to deliver its core publish/subscribe functionality. At the same time, an application building on top of Laharsub may associate any metadata it needs with a topic identifier. In particular, your stock quote application could associate a ticker symbol with a topic identifier by maintaining a (topicId, ticker) set of tuples in a database or in some static in-memory structure. This application-specific mapping can be consulted by the publisher to decide which topicId to publish messages to when a new pricing data must be distributed for a specific ticker. Similarly, clients can consult this application-specific mapping to decide which topicId should be subscribed to if the user is interested in a particular ticker symbol. In fact, if you clients are web-based, you may provide a web service running alongside Laharsub service that helps clients resolve ticker symbols to topicIds, e.g:

[ServiceOperation] 
int GetTopicIdForTicker(string ticker);

At some point I was considering an option of associating metadata with a topicId at the time of topic creation. I decided against it since I am not clear what form of the metadata would satisfy the needs of most applications. Some possibilities included a simple list of key-value pairs, a JSON structure, an XML infoset, or an oData query string. Given I don't have a clear picture of the requirements at this point, as well as the availability of an application-level workaround (as described above), I decided to postpone adding such a feature. Opinions will be appreciated.

Oct 11, 2010 at 8:12 PM

I have been using a mapping service for a few months with this product and it is working great.   

I am still trying to figure out the best way to configure Laharsub to publish stock quotes.  

Here are some of my requirements:

Each message size should be less than 500kb.  A level 2 quote has a message size of about 400kb so I can only return one at a time.  A level 1 quote has a message size of about 150kb so I should be able to return multiple quotes. A level 2 quote includes market depth and time and sales so it is bigger in size.  A level 1 quote is the best prices from market depth and the last sale so they are smaller.

I need to publish a new level 2 and level 1 quote at least every .25 seconds.  I am using a one topic per symbol per level model.  So MSFT would have a topic Id for level 2 and a seperate topic Id for level 1.  Right now I get all of the quotes from a market api and publish to laharsub using a windows service with the laharsub .net client.  I publish the level 2 and level 1 quote at the same time based on a timer every .25 seconds.

I only need the latest quote and don't need to keep a record of past quotes.  I would like to just have the latest quote in memory for each topic. 

Previously I had changed the In memory backend to only keep 1 quote and the Silverlight client to only call one quote at a time.   I downloaded the latest release and tried some different configurations, but none of them seem to capture all of these requirements.   How would you suggest I setup Laharsub to best meet these requirements?

Thanks,

Steve

Oct 11, 2010 at 9:21 PM

The "keep only the latest message published to a topic" behavior is a great feature idea - I will try to add first class support for it going forward.

In the meantime, you can try to modify the InMemoryPubSubBackend.InMemoryPublishMessageAsyncResult.AddMessageToTopic method to delete the previous message published to a topic before adding a new one, and adjust the total memory consumption counter appropriately. The new code would look something like this (note I did not even compile it, so apply apply appropriate scrutiny):

            void AddMessageToTopic(PublishedTopic publishedTopic, DateTime now, int totalSizeReduction)
            {
                lock (this.parent.syncRoot)
                {
                    this.Message.MessageId = ++this.parent.maxMessageId;
                    this.parent.currentSize += this.Message.Body.Length - totalSizeReduction;
                   
while (publishedTopic.Messages.Count > 0)
                    {
                        this.parent.currentSize -= publishedTopic.Messages[0].PubSubMessage.Body.Length;
                        publishedTopic.Messages.RemoveAt(0);
                    }

                }
                publishedTopic.Messages.Add(new PublishedMessage
                {
                    Expire = now.Add(this.parent.ttl),
                    PubSubMessage = this.Message
                });
            }

 

The section in blue removes all existing messages from the topic and adjusts the total memory storage counter appropriately. Given this method is always called under lock, there should always be 0 or 1 messages in the Messages collection. I think this change will address your scenario, let me know if it does not.