Azure Service Bus Topic to Blob read by Stream Analytics

Today, I learned that - although supposedly more efficient - you cannot actually just copy the input stream from a Azure Service Bus BrokeredMessage into a block blob and have that blob be read by Stream Analytics. This is because stream-copy introduces some control characters into the event stream, which makes it so that stream analytics cannot read the events, thinking that maybe you configured your Path Prefix Pattern wrong.


In the example above, notice how I pass CultureInfo.InvariantCulture into the path-generating DateTime expression. This is because DateTime has some magic built in, rending the expression as 2015-09-23-15 for the 23rd of September, 2015 at 15 o'clock on my machine, instead of what I (thought I) requested - 2015/09/23/15. That aside, the resulting blob had control characters and serialization data in it, which was not something I expected:




Extracting the JSON-encoded string itself and uploading that as a byte array, worked a lot better, eliminating both the control characters and the serialization information altogether:



Oh, while I have your attention, I'd like to share my Stream Analytics configuration as well, since that can be a little confusing to begin with:

Notice how I write to a blob named with a date pattern, followed by an hour and a sequence number. This corresponds to a stream analytics Path Prefix Pattern of {date}/{time}, having the Date Format be YYYY/MM/DD and hour format being HH. The BlobContainer you see in the code, is named the same thing as the general/Container configuration, so no magic here.

Alright. That should hopefully get you started (or unstuck!).

Comments

Post a Comment

Popular posts from this blog

Auto Mapper and Record Types - will they blend?

Image
TL;DR: Yes, they will. Thank you for your time. 😄 Background At Redgate, engineers enjoy a great benefit in that we get are expected to spend 10% of our time to improve on our craft.  Having suffered this week with manually having to copy a bunch of properties from a domain object to its HTTP representation, I decided to spend this week's 10% time to see if I could answer the following questions: Can use AutoMapper to simplify mapping between two types?  Does it also allow me to map between a regular type and a record type? What about the other way around? Can I use this to reduce the maintenance burden in my product? Mapping between two types [automapper fundamentals] To started and get a basic mapping done, created a .NET 5 ASP.NET Web API project, added the below package references and configured automapper : < PackageReference Include ="AutoMapper" Version ="10.1.1" /> < PackageReference Include ="AutoMapper.Extensions.Microsoft.DependencyInj
Read more

Unit testing your Azure functions - part 2: Queues and Blobs

In our last installment , we discovered how to start unit testing our Azure Functions, looking at the HTTP Trigger (and evaulating the function's HTTP response). This time, we'll take a look at using the QueueTrigger, getting supporting data from table storage and outputting Blobs. Yup, I'm tossing you into the deep end! What are we trying to accomplish? This time, we will test a function that performs some arbitrary business logic on bank accounts. The function responds to a storage queue request, logs each command and ensures that each received command is executed exactly once. Almost like a real-world function, huh? ;-) In the function above, we are introduced to three types that we need to deal with in our unit tests - the IQueryable interface, IAsyncCollector and the CloudBlockBlob concrete type. IQueryable is easy enough to work with: Introducing the Testable Async Collector When I first started exploring this topic, I figured I could just use my favourit
Read more

Testing WCF services with user credentials and binary endpoints

Image
This article discusses the following error messages that you might receive from a WCF service An error occurred when verifying security for the message HTTP/1.1 415 Cannot process the message because the content type 'application/soap+xml;charset=UTF-8;action="..."' was not the expected type 'application/soap+msbin1' When doing analysis work on existing services, invoking them in a safe test environment is a handy way of getting an understanding of their operations, especially if the documentation has gone missing (or was never written). SoapUI is a tool that can be used to test WCF Services, but did you know that Visual Studio also comes with its own set of tools? Let's start with SoapUI, which is available from  http://www.soapui.org/ . Its operations are quite straight-forward; however, you might run into issues when trying to access services that are protected by user credentials, or when accessing services with binary endpoints. I
Read more