My takeaways from A Practical Introduction To Data Science at NDC by @markawest

Data Science is a combination of computer science, machine learning and traditional research from match statistics.

It follows the scientific process: Hyptothesis and analysis. Question -> Data -> Exploration & analysis (which is about 80% of the work) -> modelling -> interpretation ("what insight am I learning from this model?") -> communication (storytelling) -> result.

A successful data science team needs a wide range of competences:
  • Whereas a data scientist uses models to try to read meaning out of the data at hand, it has generally very little experience and knowledge about operationalizing the model - putting it into production for use in systems.
  • A data engineer deals with data integration, building data driven platforms and operationalize models.
  • A visualization expert can be good at storytelling and provide insight.
  • Then, there's a process owner that deals with project management and communication.
Artificial Intelligence is a very general topic which, in turn, contain the subset Machine Learning (automatic generation of rules). Machine Learning, in turn, has a subset called Deep Learning (neural networks).

A set of training data is pushed through an Algorithm, which renders a Model (rules).

Learning modes

Supervised learning is where you categorize your data.
  • Linear regression tries to draw a line through the majority of the data points. Deviations (out of range values) are called outliers and might signal an unfit model (not enough labels?) or an error.
  • Decision trees are intuitive for software developers, since they're essentially flowcharts. The deeper your tree is, the more prone they are for overfitting (i.e. not predicting a result from your data, but matching your data to a result, since your data exactly matches previous training data).
Unsupervised learning is where a machine clusters data by its properties (i.e. all green things in one lump and all blue in another)
  • K-means clustering. Describing, not predicting. K = the amount of clusters you want the algoritm to identify.
  • Reinforcement learning is a continually learning model that learns from past mistakes.
A Feature is an input - something that I look for.
A Label is an output - a resulting value.

You need data science skills - how to select and tune algorithms - in order to be successful in this field.
Getting started: get scikit-learn and jupyter notebooks. Kaggle provides datasets. Feature engineering - massaging data - is a vital skills. Domain knowledge is key. Split dataset into training (80%) and testing (20%) sets.

Reflection: K-means clustering seems to be an applicable unsupervised learning thing for log analytics and such - to find patterns in your data.

Comments

Post a Comment

Popular posts from this blog

Auto Mapper and Record Types - will they blend?

Image
TL;DR: Yes, they will. Thank you for your time. 😄 Background At Redgate, engineers enjoy a great benefit in that we get are expected to spend 10% of our time to improve on our craft.  Having suffered this week with manually having to copy a bunch of properties from a domain object to its HTTP representation, I decided to spend this week's 10% time to see if I could answer the following questions: Can use AutoMapper to simplify mapping between two types?  Does it also allow me to map between a regular type and a record type? What about the other way around? Can I use this to reduce the maintenance burden in my product? Mapping between two types [automapper fundamentals] To started and get a basic mapping done, created a .NET 5 ASP.NET Web API project, added the below package references and configured automapper : < PackageReference Include ="AutoMapper" Version ="10.1.1" /> < PackageReference Include ="AutoMapper.Extensions.Microsoft.DependencyInj
Read more

Unit testing your Azure functions - part 2: Queues and Blobs

In our last installment , we discovered how to start unit testing our Azure Functions, looking at the HTTP Trigger (and evaulating the function's HTTP response). This time, we'll take a look at using the QueueTrigger, getting supporting data from table storage and outputting Blobs. Yup, I'm tossing you into the deep end! What are we trying to accomplish? This time, we will test a function that performs some arbitrary business logic on bank accounts. The function responds to a storage queue request, logs each command and ensures that each received command is executed exactly once. Almost like a real-world function, huh? ;-) In the function above, we are introduced to three types that we need to deal with in our unit tests - the IQueryable interface, IAsyncCollector and the CloudBlockBlob concrete type. IQueryable is easy enough to work with: Introducing the Testable Async Collector When I first started exploring this topic, I figured I could just use my favourit
Read more

Testing WCF services with user credentials and binary endpoints

Image
This article discusses the following error messages that you might receive from a WCF service An error occurred when verifying security for the message HTTP/1.1 415 Cannot process the message because the content type 'application/soap+xml;charset=UTF-8;action="..."' was not the expected type 'application/soap+msbin1' When doing analysis work on existing services, invoking them in a safe test environment is a handy way of getting an understanding of their operations, especially if the documentation has gone missing (or was never written). SoapUI is a tool that can be used to test WCF Services, but did you know that Visual Studio also comes with its own set of tools? Let's start with SoapUI, which is available from  http://www.soapui.org/ . Its operations are quite straight-forward; however, you might run into issues when trying to access services that are protected by user credentials, or when accessing services with binary endpoints. I
Read more