Slect

.NET For Apache Spark - Writing Data.

Tue, 15 Oct 2019 00:00:00 GMT

By the end of this post, you should know how to read and write from the following Spark’s core data sources:
1. Parquet.
2.JDBC/ODBC.
3. JSON
4. CSV.
5. Text

Save Modes

While writing data, we have to decide what should happen if the file we want to create already exists. The DataFrameWriter has a mode option that helps specify how Apache Spark should act if the file already exists.
Apache Spark has 4 safe modes:

|    Save Mode  |                  Description                                  |
|---------------|---------------------------------------------------------------|
|    append     | Appends the output files to the list of files                 |
|               | that already exists at that location                          |
|                                                                               |
|    overwrite  | Will completely overwrite any data that already exists there  |
|                                                                               |
|    ignore     | If data or files exist at the location, do nothing            |
|               | with the current DataFrame                                    |
|                                                                               |
| errorIfExists | Throws an error and fails the write if data or files          |
| (Default)     | already exist at the specified location                       |

You can download the Data I’m using here.

Comma Separated Value

CSV stands for Comma Separated Value. The CSV file format is very known and well supported by data processing frameworks.
In a CSV file, each line represents a single record, and commas separate each field within a record.

As we saw in the post on reading CSV files, we read CSV file using the DataFrameReader, which is accessible via the Read method of the SparkSession

var csvFile = sparkSession.Read()
              .Format("csv")
              .Options(options)
              .Schema(schemaString)
              .Load(@"path-to-file")

Now let’s see how we can save a data frame as CSV file.
To write data we use the DataFrameWriter instead of the DataFrameReader.

var sparkSession = SparkSession.Builder().GetOrCreate();

///we load the data first
var df = sparkSession.Read()
    .Option("header", "true")
    .Option("inferSchema", true)
    .Csv(@"C:\DEV\misc\retail.csv");

df.PrintSchema();
root
 |-- InvoiceNo: string (nullable = true)
 |-- StockCode: string (nullable = true)
 |-- Description: string (nullable = true)
 |-- Quantity: integer (nullable = true)
 |-- InvoiceDate: string (nullable = true)
 |-- UnitPrice: double (nullable = true)
 |-- CustomerID: integer (nullable = true)
 |-- Country: string (nullable = true)

/// Now let's filter invoices from Germany

var de = df.Where("Country == 'Germany'");

/// Let's save the result

de.Write().Format("CSV")
    .Option("header", true)
    .Option("nullValue","NULL") // Use the string "NULL" for null value
    .Option("path", @"C:\DEV\misc\de-invoices.csv")
    .Mode(SaveMode.Overwrite) // Overwrite existing file
    .Save();

The save operation creates a folder named de-invoices.csv and depending on the number of partitions the data frame has, the folder contains many part files (part-00000-…), and each part file represents the content of a single partition.

There are many options we can use when saving CSV files: header : add the header to every part file. sep : set a single character as a delimiter for each field and value. nullValue: sets the string representation of a null value. you can take a look at all available options for writing and reading csv file in the documentation

Apache Parquet.

Apache Parquet is a free and open-source column-oriented data storage format supported by many data processing frameworks. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk.

It provides columnar compression, which saves storage space and allows for reading individual columns instead of entire files. Reading and Writing from Parquet is more efficient than CSV or JSON. Apache Parquet supports complex data types, which are not supported by CSV.

Apache Parquet is the default data source for Apache Spark

Parquet files encode the schema together with the data; therefore, the format of the data is always known. Parquet also supports schema evolution, so that we can evolve the schema of a file over time. Schema evolution means we can add or remove columns to file over time without losing data.

To save more space, we can set the compression while storing parquet files. The following compression codecs are supported:
none, uncompressed, snappy, gzip, low, brotli, lz4, and zstd.

Let’s store our previously data frame as parquet file

de.Write()
    .Option("compression","gzip") // set compression to gzip. Default is snappy
    .Mode(SaveMode.Overwrite)
    .Parquet(@"C:\DEV\misc\de-invoices.parquet");

The code above creates a folder named de-invoices.parquet that contains one or many parquet files depending on the number of partitions. In my case, it contains one file that is 90Kb large. That is a huge saving compared to the 730kb large csv file stored previously.

We can easily read parquet files as follow :

// No need to specify format, because parquet is default format
var parquetDf = sparkSession.Read()
    .Load(@"C:\DEV\misc\de-invoices.parquet");

// Specify the format
var parquetDf = sparkSession.Read()
    .Format("parquet")
    .Load(@"C:\DEV\misc\de-invoices.parquet");

// Use the Parquet method
var parquetDf = sparkSession.Read()
    .Parquet(@"C:\DEV\misc\de-invoices.parquet");

JDBC/ODBC

In this previous post, I wrote about how to connect and read from a JDBC/ODBC data source using PostgreSQL.

Building on that post, I’ll now show you how to write a data frame to a JDBC source.

First, let’s set up all we need to connect to RDBMS using Apache Spark

var hostname = "localhost";
var portname = 5433;
var databaseName = "dvdrental";

var dbUser = "spark_user";
var dbUserPwd = "sparkuser";

var jdbcUrl = $"jdbc:postgresql://{hostname}:{portname}" +
    $"/{databaseName}?" +
    $"user={dbUser}&" +
    $"password={dbUserPwd}";

Now let’s read some data, change it, and write it back.

//Read data from a table named actor.

var jdbcDF = sparkSession.Read().Format("jdbc")
            .Option("url", jdbcUrl)
            .Option("dbtable", "public.actor")
            .Load();

jdbcDF.PrintSchema();
root
 |-- actor_id: integer (nullable = true)
 |-- first_name: string (nullable = true)
 |-- last_name: string (nullable = true)
 |-- last_update: timestamp (nullable = true)

jdbcDF.Show();
+--------+----------+------------+--------------------+
|actor_id|first_name|   last_name|         last_update|
+--------+----------+------------+--------------------+
|       1|  Penelope|     Guiness|2013-05-26 14:47:...|
|       2|      Nick|    Wahlberg|2013-05-26 14:47:...|

 // Create a new data frame and change the first_name column
 // Concatenate first_name and actor_id

var newDF = jdbcDF.WithColumn("first_name",
                            Functions.ConcatWs("_", jdbcDF.Col("first_name"),
                            jdbcDF.Col("actor_id")))
                    .SelectExpr("first_name", "last_name")
                    .Limit(10);
newDF.PrintSchema();
root
 |-- first_name: string (nullable = false)
 |-- last_name: string (nullable = true)

 newDF.Show();
 +------------+------------+
|  first_name|   last_name|
+------------+------------+
|  Penelope_1|     Guiness|
|      Nick_2|    Wahlberg|
|        Ed_3|       Chase|

Note that the newly created data frame is not yet available in the data source, so if I query the database for the actor Penelope_1, the result set would be empty.
Now let’s write the data frame to the database

// First specify database connection arguments
var writeProps = new Dictionary<string, string> {
    {"driver", "org.postgresql.Driver"}
};

// Then save the content of data frame to the database
newDF.Write()
    .Mode(SaveMode.Append)
    .Jdbc(jdbcUrl, "public.actor", writeProps);

To get the new data, we create a data frame that contains only first names with an underscore.

var df = sparkSession.Read()
    .Option("url", jdbcUrl)
    .Option("dbtable", "public.actor")
    .Format("jdbc")
    .Load()
    //Filter for first_name containing an underscore
    .Where("first_name like '%\\_%'");

df.PrintSchema();
root
 |-- actor_id: integer (nullable = true)
 |-- first_name: string (nullable = true)
 |-- last_name: string (nullable = true)
 |-- last_update: timestamp (nullable = true)

df.Show();
+--------+------------+------------+--------------------+
|actor_id|  first_name|   last_name|         last_update|
+--------+------------+------------+--------------------+
|     201|  Penelope_1|     Guiness|2019-10-09 06:13:...|
|     202|      Nick_2|    Wahlberg|2019-10-09 06:13:...|
|     203|        Ed_3|       Chase|2019-10-09 06:13:...|

Text Files

To write text files, use the Text() method.
Note that if the data frame has more than 1 column, the operation fails.

df.Select("first_name")
    .Write()
    .Mode(SaveMode.Overwrite)
    .Text(@"C:\DEV\misc\first_name.txt");

JSON Files

Writing JSON files is as simple as reading them.
As with other files format, we get one file per partition, and the entire data frame is written out as a folder.

df.Write().Json(@"C:\DEV\misc\person.json");

//The resulting JSON file looks like this:
{"actor_id":201,"first_name":"Penelope_1","last_name":"Guiness","last_update":"2019-10-09T06:13:54.676+02:00"}
{"actor_id":202,"first_name":"Nick_2","last_name":"Wahlberg","last_update":"2019-10-09T06:13:54.676+02:00"}

References:
https://parquet.apache.org/
https://databricks.com/glossary/what-is-parquet
https://spark.apache.org/docs/latest/sql-data-sources.html

Apache Spark - Architecture Part 1: Driver and Executors

Tue, 24 Sep 2019 00:00:00 GMT

By the end of this post you should know how to do the following:
1. Define the different cluster management systems Spark can run on.
2. List the major components of Apache Hadoop YARN
2. Explain the relationship between all major components of YARN
3. Explain the functionalities of the Driver an Executors in Apache Spark.
4. Explain how Spark uses cluster resource to process Big-Data

Cluster

The computing resources (i.e., Ram, disk, network) needed to efficiently store and process Big-Data can’t fit on a single machine. We need not only a large amount of disk space to store all the data, but all computation on the data must happen as fast as possible, and to achieve this goal we use a cluster.

When a group of loosely or tightly connected computers work together so that they are viewed as if they are one computer, it is called cluster.

The components of a cluster are connected through fast networks, and this makes communication and data transfer inside the cluster as efficient as possible.
A cluster is a pool of different resource types like:

DISK
CPU
GPU
BANDWIDTH
MEMORY
IP-ADDRESS

The image below shows an illustration of a cluster

On a single computer, the operating system is responsible for the resource management; it has the global (local) view of all available resources, it decides how much of any resource type an application can use and how much of it is available to run other applications.
However, how does resource management works when we have thousands of computers working together as a unit?

Cluster Management System

To manage a cluster of compute instances, we need a cluster management system.
A cluster management system provides the mechanisms through which you interact with the resources available in your cluster.

A cluster management system tracks resource allocation, scale based on resource utilization, check and monitor the health of individual nodes (computer in a cluster) or resource.

Example of cluster management systems are:

As of this writing, Apache Spark [version 2.4.4], can run on top of Kubernetes, Mesos, and Apache Hadoop YARN.

I’ll use Apache Hadoop YARN to explain how Apache Spark runs on a cluster.

Apache Hadoop YARN (Yet Another Resource Negotiator)

YARN can be installed on a set of heterogeneous hardware to build a cluster, and it has four main components:

ResourceManager - RM

The ResourcesManager runs as a daemon (process) on a dedicated machine inside the cluster.

The ResourceManager schedule and arbitrates resources among all distributed applications running in the cluster.

When we submit an application to the cluster, we also tell the resource management system how much of each resource types our application needs to run well; the resource management system then find a set of nodes in the cluster with enough resources to satisfies our requirements; it then allocates the number of resources required and assign them to our application.

The process of matching tasks to available compute resources at a given time is called scheduling.

YARN Containers

The ResourceManager allocates resource to run an application in the form of containers.
A container is a basic unit of allocation in YARN.

YARN Containers are fine-grained resource allocation across multiple resource types such as memory, CPU, disk, network, GPU.
A node in the cluster can run multiple resource containers, so on a node, we can have one container with 3Gb Ram,2 Cpu Core and another container with 512 Mb and 1 Cpu Core.

A container runs on a node that’s managed by a NodeManager.
A container makes use of resources assigned to it on a node to perform computation tasks;
A container depends on some application code or shared libraries that are represented as local resources.

NodeManager - NM

On each node in the cluster, there is another process running called the NodeManager.

The primary goal of the NodeManager is to manage application containers assigned to it by the ResourceManager.

The NodeManager is the “worker” process in YARN, and it takes care of the individual node in the cluster. Some responsibilities of the NodeManager are:

Launch containers to perform actual computation work.
Monitor the health of the physical node and reports it back to the ResourceManager
Monitor resource availability on each node.
Monitor resource utilization of individual containers.
Track node health and reports it to the central ResourceManager.
Manage the lifecycle of a container (e.g., starting, killing).

A container is monitored by the NodeManager and scheduled by the ResourceManager.
The central ResourceManager and the collection of NodeManagers create the unified computational infrastructure of the cluster.

ApplicationMaster - AM

The ApplicationMaster is the process that coordinates the execution of an application inside the cluster.

The ApplicationMaster contains the user code.
Each instance of an application has its own unique ApplicationMaster.
Some Tasks of the ApplicationMaster are:

Computing the resource requirements of the application.
Negociate resources in the form of containers from the ResourcesManager.
Using allocated containers by working with the NodeManagers.
Manage all lifecycle aspects of a job, including dynamically increasing and decreasing resources consumption.
Reacting to container or node failures by requesting alternative resources from the ResourceManager if needed.
Tracking the status of running containers and monitoring their progress.

The ApplicationMaster runs in the cluster just like any other container, and each application starts as an ApplicationMaster. Once started, the ApplicationMaster must negotiate with the ResourceManager for more containers depending on the application needs.

Enter The Spark Cluster

A Spark Cluster is made of a Spark-Driver and a set of Executors.

The Spark-Driver

With all we’ve learned till now it’s easier to tell what the Spark-Driver is:
The Spark-Driver is the ApplicationMaster process of a Spark application.
The Spark-Driver requests resources from the ResourceManager and works together with the NodeManagers to executed and monitor tasks.

The Spark-Executors.

Spark-Executors are processes running as containers that perform computation tasks assigned by the Spark-Driver.

Spark-Executors have one core responsibility: take the tasks assigned by the driver, run them, and report their state (success or failure) and results back to the Spark-Driver.

After Spark-Submit

The following is what happens when a Spark-Application is submitted to a YARN cluster:

The Spark-Application is submitted to the ResourceManager
The Spark-Driver is started and registers with the ResourceManger to later report about the application status.
The Spark-Driver requests Spark-Executors from the ResourceManager to perform computation.
The Spark-Executors assigned to the Spark-Driver are presented to the NodeManager. The NodeManager then launches the Executors on the nodes.
Spark-Executors are responsible for the computation. They keep in contact with the Spark-Driver to report their progress.
When the application is complete, the Spark-Executors are stopped, and the Spark-Driver is unregistered from the ResourceManager

Conclusion

We examine how a cluster management system works together with a cluster computing framework for resource scheduling.

I use Apache Hadoop YARN to explain the concept, but note that all other cluster management system use similar model just with different implementation and terminology, for example, the resource manager component for Mesos is called Mesos master.

A Spark cluster is a combination of a Spark ApplicationMaster and the containers it needs to performs computation.
The ApplicationMaster for Spark applications is called the Driver.
The Container executing Spark Tasks are called Executors.

References:
https://www.massey.ac.nz/~mjjohnso/notes/59735/seminars/04244354.pdf
https://en.wikipedia.org/wiki/Computer_cluster
https://www.oreilly.com/ideas/a-tale-of-two-clusters-mesos-and-yarn
https://github.com/Larry3z/HadoopRelatedBooks/blob/master/Apache%20Hadoop%20YARN%20Arun%20C.%20Murthy%20Addison-Wesley%20March%202014.pdf

.Net for Apache Spark - DataFrame, Part 4: Complex Types

Sun, 01 Sep 2019 00:00:00 GMT

By the end of this post you should know how to do the following:
1. List the different complex types available in Apache Spark
2. How to query a DataFrame that has complex datatype

Apache Spark has three different complex type: Arrays, Structs and Maps.

The file I’m using for this post contains data describing medical procedures.
Each procedure has an ID and a Description:

root
 |-- ProcId: integer (nullable = true)
 |-- Description: string (nullable = true)

+------+---------------------------------------------------------------------+
|ProcId|Description                                                          |
+------+---------------------------------------------------------------------+
|1     |Dilation of Left Hand Artery, Bifurcation, with Drug-eluting Intri...|
|2     |Insertion of Radioactive Element into Cervix, Via Natural or Artif...|
+------+---------------------------------------------------------------------+

Array

First, I create an array out of the Description column using the Split method. The Split method takes a string and split it by a given regular expression; in our case, it takes the contents of the Description column and returns an array of words.

using Microsoft.Spark.Sql;

var df = sparkSession.Read()
          .Option("header", "true")
          .Option("inferSchema",true)
          .Csv("file.csv");
var splitted = df.Select(df.Col("ProcId"),Functions.Split(df["Description"], " ").Alias("arr_col"));

splitted.show(5,0,false)
+------+--------------------------------------------------------------------------------------------------------------------------------+
|ProcId|arr_col                                                                                                                         |
+------+--------------------------------------------------------------------------------------------------------------------------------+
|1     |[Dilation, of, Left, Hand, Artery,, Bifurcation,, with, Drug-eluting, Intraluminal, Device,, Percutaneous, Endoscopic, Approach]|
|2     |[Insertion, of, Radioactive, Element, into, Cervix,, Via, Natural, or, Artificial, Opening, Endoscopic]                         |
+------+--------------------------------------------------------------------------------------------------------------------------------+

Apache Spark offers many functions, and here I show you some useful functions for arrays. For .NET you can find most of these functions in the Microsoft.Spark.Sql namespace

We can select a value from an array using it zero-based index

splitted.SelectExpr("ProcId", "arr_col[2]").Show();

+------+---------------+
|ProcId|     arr_col[2]|
+------+---------------+
|     1|           Left|
|     2|    Radioactive|

We can get the size of the array with the Size method. We can see that the first array contains 13 words.

splitted.Select(splitted.Col("ProcId"),Functions.Size(splitted.Col("arr_col"))).Show();

+------+-------------+
|ProcId|size(arr_col)|
+------+-------------+
|     1|           13|
|     2|           12|

With the method ArrayContains we can check if an array contains a given value. The ArrayPosition method returns the position of the first occurrence of the value in the given array. We can see that the string Radioactive is at the third position in the array. Note that the position is not zero-based, but 1 based index

splitted.Select(splitted.Col("ProcId"), Functions.ArrayContains(splitted.Col("arr_col"), "Radioactive")).Show();

+------+------------------------------------+
|ProcId|array_position(arr_col, Radioactive)|
+------+------------------------------------+
|     1|                                   0|
|     2|                                   3|
|     3|                                   0|

Struct

A struct is a complex type that represents a single column with multiple attributes.
You can think of a struct as a Dataframe within a DataFrame.
For Example, let’s create a column containing the first two words of the Description column. To create a struct, wrap columns in parenthesis in a query or use the Functions.Struct method. We create a column named firstTwo with two fields.

var structDF = splitted.SelectExpr("*", "(arr_col[0] first,arr_col[1] second) as firstTwo");

structDF.PrintSchema();
root
 |-- ProcId: integer (nullable = true)
 |-- arr_col: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- firstTwo: struct (nullable = false)
 |    |-- first: string (nullable = true)
 |    |-- second: string (nullable = true)

structDF.Show();
+------+--------------------+--------------------+
|ProcId|             arr_col|            firstTwo|
+------+--------------------+--------------------+
|     1|[Dilation, of, Le...|      [Dilation, of]|
|     3|[Bypass, Duodenum...|  [Bypass, Duodenum]|
|     4|[Excision, of, Ce...|      [Excision, of]|

The PrintSchema method shows us that of the newly created column is of type struct.
We can query a struct using the dot notation or the GetField Method

// Select all field of the struct
structDF.Select("*").Show();
+------+--------------------+--------------------+
|ProcId|             arr_col|            firstTwo|
+------+--------------------+--------------------+
|     1|[Dilation, of, Le...|      [Dilation, of]|
|     2|[Insertion, of, R...|     [Insertion, of]|

//Select a field using the dot-Notation
structDF.Select("procid", "firstTwo.first").Show();
+------+-----------+
|procid|      first|
+------+-----------+
|     1|   Dilation|
|     2|  Insertion|

//select a field using the GetField method
structDF.Select(structDF.Col("firstTwo").GetField("second")).Show();

+------+---------------+
|procid|firstTwo.second|
+------+---------------+
|     1|             of|
|     2|             of|

Maps

a map represent a set of key-value pairs. The keys must be scalar type, and the values can be of any type, including complex types like array,struct, or map.

we can create with the help of the Map function. For example, let’s create a map that uses the column procid as key and the description column as value.

var mapDF = df.Select(df.Col("ProcId"), Functions.Map(df.Col("procid"), df.Col("description")).Alias("map_col"));
mapDF.PrintSchema();
root
 |-- ProcId: integer (nullable = true)
 |-- map_col: map (nullable = false)
 |    |-- key: integer
 |    |-- value: string (valueContainsNull = true)

mapDF.Show(2,40,false);
+------+----------------------------------------+
|ProcId|              map_col                   |
+------+----------------------------------------+
|     1|[1 -> Dilation of Left Hand Artery, B...|
|     2|[2 -> Insertion of Radioactive Elemen...|

We can query the map by using the keys, and if the key is invalid, the value is null.
You can also create a DataFrame out of the map using the Explode method.

mapDF.SelectExpr("map_col[1]").Show();

//there is only one row for the key "1".
+--------------------+
|          map_col[1]|
+--------------------+
|Dilation of Left ...|
|                null|
|                null|

var explodeMapDF = mapDF.Select(Functions.Explode(mapDF.Col("map_col")));

explodedMapDF.PrintSchema();
root
 |-- key: integer (nullable = false)
 |-- value: string (nullable = true)

 explodedMapDF.Show();
+---+--------------------+
|key|               value|
+---+--------------------+
|  1|Dilation of Left ...|
|  2|Insertion of Radi...|

.Net for Apache Spark - DataFrame, Part 3: Basic Transformations

Sun, 25 Aug 2019 00:00:00 GMT

This post introduces you to the fundamentals of working with DataFrames.
By the end of this post you should know how to do the following with a Dataframe:
1. Select specific columns.
2. Filter rows.
3. Add a Column.
4. Remove a Column.
5. Rename a Column.
6. Change the Datatype of a Column.

Select Columns

There are two methods to extract the column’s values of a DataFrame: Select and SelectExpr.

The Select method takes the name of a column and returns its value. The Select method can also take a Column object as an argument.

Given a DataFrame with the following schema

 |-- invDate: string (nullable = true)
 |-- invItemId: string (nullable = true)
 |-- invWareHouseId: string (nullable = true)
 |-- invQuantityOnHand: string (nullable = true)

We can extract data from the DataFrame using the following query:

// Extract data using the name of a column
var res = df.Select("invDate","invQuantityOnHand");

// Extract data using column objects
var res = df.Select(df["invDate"], df.Col("invQuantityOnHand"));

note that you can’t mix strings and column object as argument, so a call like the below is invalid:

var res = df.Select(df["invDate"],"invQuantityOnHand");

The SelectExpr takes an SQL expression as an argument, and as long as the columns resolve, the expression is valid. With the SelectExpr it is possible to specify aggregate over the entire DataFrame.

// Expression and the name of column are the same
var res = df.SelectExpr("invDate");

// Expression checks the value of a column
var res = df.SelectExpr("invItemId as ItemID","invQuantityOnHand = 0 as Available");

res.Show();
+------+---------+
|ItemID|Available|
+------+---------+
|  2122|     true|
|  2125|    false|
|  2126|    false|

// Aggregation over the entire DataFrame
var res = df.SelectExpr("count(invItemId) as TotalItems", "avg(invQuantityOnHand) as AverageItemsQuantity");

res.Show();
+----------+--------------------+
|TotalItems|AverageItemsQuantity|
+----------+--------------------+
|       998|   516.5397489539749|
+----------+--------------------+

Now we know how to extract data from the columns of a DataFrame. Next, we learn how to return only specific rows from a DataFrame.

Filter Rows

There are two methods to filter a DataFrame: Where and Filter. Both methods allow you to specify a logical expression to filter rows from a DataFrame. Only rows for which the expression evaluates to TRUE are returned. The query below creates a new DataFrame that contains only Product with more than 500 items available.

/// Get only product with more than 500 item available
var res = df.Where("invQuantityOnHand > 500");

Note that it’s not useful to put multiple filters in the same expression, because Apache Spark automatically performs all filtering operations at the same time. To specify multiple filters, chain them sequentially:

// Where and Filter also work with column
var res = df.Filter(df["invQuantityOnHand"] > 500)
    .Where(df["invItemId"] < 3000);

Add Column

Adding a column to a DataFrame means that we create a DataFrame out of an existing one and add an extra column to the new one.

To add a new column, use the WithColumn method.
For example, let’s add a new column to our DataFrame to see whether an item is available or not.

var res = df.WithColumn("ItemAvailable", df.Col("invQuantityOnHand") > 0 );

res.PrintSchema();
 |-- invDate: string (nullable = true)
 |-- invItemId: string (nullable = true)
 |-- invWareHouseId: string (nullable = true)
 |-- invQuantityOnHand: string (nullable = true)
 |-- ItemAvailable: boolean (nullable = true)

res.Show(2);
+-------+---------+--------------+-----------------+------------- +
|invDate|invItemId|invWareHouseId|invQuantityOnHand|ItemAvailable |
+-------+---------+--------------+-----------------+------------- +
|2450815|     2122|             1|              688|          true|
|2450815|     2125|             1|              119|         false|
+-------+---------+--------------+-----------------+--------------+

Rename Column

To rename a column use the WithColumnRenamed method.
For example let’s rename the column invQuantityOnHand to ItemCount

var res = df.WithColumnRenamed("invQuantityOnHand", "ItemCount");

res.PrintSchema();
 |-- invDate: string (nullable = true)
 |-- invItemId: string (nullable = true)
 |-- invWareHouseId: string (nullable = true)
 |-- ItemCount: string (nullable = true)

Remove Column

When we use the Select method, we can specify only the columns we want. Therefore we can use the Select method to remove columns from a DataFrame. However, Apache Spark has a method for dropping columns.
The Drop method allows you to remove one or more columns from a DataFrame.

var res = df.Drop("invWareHouseId", "invDate");

res.PrintSchema();
 |-- invItemId: string (nullable = true)
 |-- invQuantityOnHand: string (nullable = true)

Change Type

Suppose we want to change the datatype of the column invWareHouseId from string to int. We can do this by adding a new column containing the value of invWareHouseId as int instead of string and then drop the invWareHouseId column.
To change the type of column, use the Cast method.

var res = df.WithColumn("WareHouseId", df.Col("invWareHouseId").Cast("int"))
    .Drop("invWareHouseid");

res.PrintSchema();
 |-- invDate: string (nullable = true)
 |-- invItemId: string (nullable = true)
 |-- invQuantityOnHand: string (nullable = true)
 |-- WareHouseId: integer (nullable = true)

The DataFrames we used thus far, all contains simple type like string and int, but this is not always the case. Apache spark also has support complex type, i.e., Struct and Maps. In the next post, I show you how to work with DataFrames that contain columns with complex types

.Net for Apache Spark - DataFrame, Part 2: Transformations

Sun, 18 Aug 2019 00:00:00 GMT

In Apache Spark, business logic is expressed using Transformations.
This blog post is a theoretical introduction to DataFrame Transformation

A DataFrame is immutable, that means they can’t be changed once created.
We can only change A DataFrame by instructing Apache Spark to transform it into a new DataFrame.

A Transformation is an instruction that tells Apache Spark how to modify a DataFrame. We create a new DataFrame by applying transformations on an existing DataFrame.

For example, given a JSON file with the following schema:

root
 |-- city: string (nullable = true)
 |-- first_name: string (nullable = true)
 |-- id: long (nullable = true)
 |-- last_name: string (nullable = true)

we can load and filter it to create a new DataFrame that contains only cities with the name starting with ‘S’ by applying the filter transformation:

var df = sparkSession.Read()
    .Json(@"Path-to-file.json");
var cityStartingWith_S = dataFrame.Filter("city like 'S%'");

A Transformation does not trigger any computation.
Apache Spark compiles all transformations of a DataFrame; by doing this, Spark can optimize the data flow and make it more efficient to run in a distributed environment.

To perform work in parallel, Spark breaks up data into chunks, called partitions.
A partition is a collection of rows that sit on one physical machine in our cluster. A DataFrame partition represents how the data is distributed physically across our cluster during execution.

Apache Spark has two types of transformations:

Narrow transformation: where one input partition contribute to only one output partition ( 1 to 1)
Wide transformation: where one input partitions contribute to many output partitions (1 to n)

An action triggers a computation in Apache Spark.
An action instructs Spark to compute the result from a series of transformations.
Apache Spark has three kinds of actions:

Actions to view data in the console
Actions to collect data to native objects in the respective language like Scala or Python
Actions to write to output data sources such as file system or RDBMS

Sources: https://pages.databricks.com/gentle-intro-spark.html

.Net for Apache Spark - DataFrame, Part 1: Fundamentals

Sun, 11 Aug 2019 00:00:00 GMT

By the end you should know how to do the following:
1. Describe what a DataFrame is in Apache Spark.
2. Explain what row and columns are.

When we use the read method of the DataFrameReader to load files, the result is a DataFrame, which is an abstraction that represents a distributed collection of data.

When working on a local machine, the DataFrame resides only on one machine and on a cluster the DataFrame is distributed across all machines of the cluster.

A DataFrame is similar to a table in a traditional database, it represents a table of data, and you can think of it as a spreadsheet that resides on your local machine or distributed across all machines of your cluster

Like a spreadsheet, a DataFrame has rows and columns.

A row represents a record of data inside the DataFrame.
A row has a fixed number of columns.
A column has a name and a type like string or array, that must be the same for every row of the DataFrame.

A DataFrame has a Schema that defines the column names and types of the DataFrame.
You can define a Schema manually (recommended) or let Apache Spark infer it from the data you are reading.

Each row or record in a DataFrame is of type Row therefore a DataFrame is a collection of type Row.

The Row Type is Apache Spark internal in-memory format for highly specialized and efficient computation.

Now let’s say we have a json file, the following code returns a DataFrame, that contains the file

var df = sparkSession.Read()
    .Json(@"Path-to-file.json");

We can inspect the schema of the DataFrame using the PrintShema() method.

df.PrintSchema();

The Schema shows us the name and type of all columns available in the DataFrame

root
 |-- email: string (nullable = true)
 |-- first_name: string (nullable = true)
 |-- friends: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- first_name: string (nullable = true)
 |    |    |-- last_name: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- id: long (nullable = true)
 |-- ip_address: string (nullable = true)
 |-- last_name: string (nullable = true)
 |-- workplace: struct (nullable = true)
 |    |-- friday: string (nullable = true)
 |    |-- monday: string (nullable = true)

.Net for Apache Spark - Loading Data, Part 3 : RDBMS

Sun, 04 Aug 2019 00:00:00 GMT

By the end of this post, you should know how to do the following:
1. How to connect to SQL databases using JDBC .
2. Specify a query that will be used to read data into Apache Spark
3. Set a custom schema to use for reading data from a JDBC data source

var hostname = "localhost";
var portname = 5433;
var databaseName = "dvdrental";

var dbUser = "spark_user";
var dbUserPwd = "sparkuser";

var jdbcUrl = $"jdbc:postgresql://{hostname}:{portname}" +
    $"/{databaseName}?" +
    $"user={dbUser}&" +
    $"password={dbUserPwd}";

var sparkSession = SparkSession.Builder().GetOrCreate();

var jdbcDF = sparkSession.Read().Format("jdbc")
    .Option("url", jdbcUrl)
    .Option("dbtable", "public.actor")
    .Load();

The above snippet is all you need to read data from a JDBC datasource, in this case, a PostgreSQL database.

To read Data from a JDBC Source do the following:

Set the input data source format to jdbc by using the Format method of the DataFrameReader class.
Set the data source to read from using the option url and pass a valid JDBC connection URL.
Specify the table to read from with the dbtable option and the table.
Use Load() to import the data

Apache Spark will use all the data in the specified table for any operation we want to do with data. If it is a large table, it’s better to specify a query to read the data so that we only use a relevant set.

Specify a query

Use the query option together with a valid SQL statement. For example, we could use the following query to only load actors with the last name starting with the character ‘B’

var jdbcDF = sparkSession.Read().Format("jdbc")
    .Option("url", jdbcUrl)
    .Option("query","select * from public.actor where last_name like '%B%'")
    .Load()

Note that you can’t use the option query and dbtable at the same time.

Specify a custom schema.

the current schema of the test data is like this:

root
 |-- actor_id: integer (nullable = true)
 |-- first_name: string (nullable = true)
 |-- last_name: string (nullable = true)
 |-- last_update: timestamp (nullable = true)

and here an excerpt of the data:

+--------+-----------+---------+--------------------+
|actor_id| first_name|last_name|         last_update|
+--------+-----------+---------+--------------------+
|      12|       Karl|    Berry|2013-05-26 14:47:...|
|      14|     Vivien|   Bergen|2013-05-26 14:47:...|

I will use a custom schema to change the data type of actor__id from int to long and the data type of last__update from timestamp to date.

To do to this I use the customSchema option :

var jdbcDF = sparkSession.Read().Format("jdbc")
 .Option("url", jdbcUrl)
 .Option("customSchema","actor_id bigint,last_update date")
 .Option("query","select * from public.actor where last_name like '%B%' limit 10")
 .Load();

after applying the changes the schema we can see the changes from int to long and from timestamp to date.

root
 |-- actor_id: long (nullable = true)
 |-- first_name: string (nullable = true)
 |-- last_name: string (nullable = true)
 |-- last_update: date (nullable = true)

a look at the data shows that the changes were successful:

+--------+-----------+---------+-----------+
|actor_id| first_name|last_name|last_update|
+--------+-----------+---------+-----------+
|      12|       Karl|    Berry| 2013-05-26|
|      14|     Vivien|   Bergen| 2013-05-26|

Watch the video below to see this in action.

.Net for Apache Spark - Loading Data, Part 2 : JSON files

Tue, 30 Jul 2019 00:00:00 GMT

By the end of this post, you should know how to do the following:
1. Read JSON files.
2. Configure options for the JSON format.
3. Specify a complex DDL-formatted schema for a JSON file.
4. Handle file with corrupted records

Single Line Mode

var sparkSession = SparkSession.Builder().GetOrCreate();
var jsonFile = sparkSession.Read()
                            .Json(@"path-to-json-file.json");

The above snippet is all you need create a Dataframe from a JSON file, that contains a valid JSON record or object per line:

{"id":1,"first_name":"Aurie","last_name":"Mowne","birthday":"11-05-1989"}
{"id":2,"first_name":"Lucio","last_name":"Heasley","birthday":"13-11-2003"}

Apache Spark will automatically infer the schema of the file, the schema of a file containing the above JSON record looks like this:

root
 |-- birthday: string (nullable = true)
 |-- email: string (nullable = true)
 |-- first_name: string (nullable = true)
 |-- id: long (nullable = true)

The field birthday is a date field but the inferred data type is of type string. To fix this error, we must specify the schema manually:

var jsonFile = sparkSession.Read()
    .Schema("id BIGINT, first_name STRING, last_name STRING, birthday date")
    .Json(@"path-to-json-file.json");

now the correct schema looks like this:

root
 |-- id: long (nullable = true)
 |-- first_name: string (nullable = true)
 |-- last_name: string (nullable = true)
 |-- birthday: date (nullable = true)

now that the field birthday is set up correctly let’s take a look at the data with the show() method of the DataFrame class

+---+----------+----------+----------+
| id|first_name| last_name|  birthday|
+---+----------+----------+----------+
|  1|     Aurie|     Mowne|0016-10-09|
|  2|     Lucio|   Heasley|0019-04-26|
|  3|   Hayward|      Cope|0029-07-29|

as you can see, although we defined the correct data type for the field birthday, the provided string wasn’t parsed correctly. The reason being that, the default date format for spark is yyyy-MM-dd and the string for the birthday field is dd-MM-yyyy, and of course, there is an option to specify the date format

var jsonFile = sparkSession.Read()
  .Schema("id BIGINT, first_name STRING, last_name STRING, birthday date")
  .Option("dateFormat", "dd-MM-yyyy")
  .Json(@"path-to-json-file.json")

now if we inspect the file again, we can see that the birthday is now correct

+---+----------+----------+----------+
| id|first_name| last_name|  birthday|
+---+----------+----------+----------+
|  1|     Aurie|     Mowne|1989-05-11|
|  2|     Lucio|   Heasley|2003-11-13|

Multiple Line Mode

Suppose you have a JSON file where a JSON object can span multiple lines

[
  {
    "id": 1,
    "first_name": "Obediah",
    "last_name": "Castane",
    "email": "ocastane0@google.ca",
    "gender": "Male",
    "ip_address": "1.79.179.139",
    "friends": [
      { "first_name": "Harbert", "last_name": "Dybell" },
      { "first_name": "Wallie", "last_name": "Muge" },
      { "first_name": "Lynette", "last_name": "Rivel" },
      { "first_name": "Ignaz", "last_name": "Cumberlidge" }
    ],
    "workplace": {
      "monday": "71690 Gina Pass",
      "friday": "839 Jackson Point"
    }
  },
  {....},
  {....}
]

To load this file, we must enable the multiline option. With this option, the file will be loaded as a whole entity and cannot be split. Note how the file is enclosed with [] brackets; this is important because otherwise, the load may fail.

var jsonFile = sparkSession.Read()
              .Option("multiLine", true)
              .Json(@"path-to-json-file.json");

Now let see how to specified a schema for a JSON file like this one.

var schemaString = "id BIGINT, first_name STRING,"+
    "last_name STRING,"+
    "email STRING, gender STRING, ip_address STRING,"+
    "friends ARRAY<STRUCT<first_name:STRING,last_name:STRING>>,"+
    "workplace STRUCT<friday:STRING,monday:STRING>";

to learn more about how to specify a complex DDL, visit the Hive Language Manual

Handling corrupted records

The parser can run in 3 different mode

PERMISSIVE (default): nulls are inserted for fields that could not be parsed DROPMALFORMED: drops lines that contain fields that could not be parsed FAILFAST: aborts the reading if any malformed data is found

the mode in which the parser runs can be set using the mode option

var jsonFile = sparkSession.Read()
    .Schema(schemaString)
    .Option("mode","dropmalformed")

Remember that when a multiline file is read, the file is processed as a whole and can’t be split, therefore if one corrupted record is found during parsing, the whole process may fail. Watch the video below to see this in action.

.Net for Apache Spark - Loading Data, Part 1 : CSV files

Thu, 25 Jul 2019 00:00:00 GMT

By the end of this post, you should know how to do the following:
1. Read csv file using format() and load().
2. Configure options for the csv format.
3. Specify a DDL-formatted schema for a csv file.

Load data using format() and load()

Let’s start with a file that contains character separated values. The most common form of a character separated file is CSV: Comma Separated Value. In this example, we will use a file with values separated by a vertical bar |.

The file represents the inventory of a fictive retailer, each row in the file represents the quantity of a particular item on-hand at a given warehouse during a specific week. The file contains four values separated by a vertical bar, and here is the definition of the fields:

The id of the date
The id of the warehouse item
The id of the warehouse that holds the item
The inventory quantity of the item at hand in the given warehouse

To read and load the file, we use the Read method of the spark session; the spark session is the entry point into all functionality in Spark.

so, in our app, we first need to create a spark session if we want to do anything. The Read method returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.

On the DataFrameReader, we then call the method Format to specify the format of the file we want to load, supported formats are json, parquet, jdbc, orc, libsvm, csv, text.

sparkSession.Read()
            .Format("csv") ...

In this example I’m loading a character separated file, so I have to tell Spark which character is used inside as value separator, this character is also called the delimiter.

To specify the delimiter, we use the Options method of DataFrameReader; the Options method takes a Dictionary<string,string> as a parameter.

 sparkSession.Read()
    .Format("csv")
    .Options(new Dictionary<string, string>
            {
                { "delimiter", "|" }
            }
            ) ...

By default Spark will use the datatype string for all fields in the file, The string datatype may be enough in cases you want to inspect the file.

There are two ways to set the schema of a file. The first one is to let Spark infer the schema of the data, by setting the inferSchema option to true

 sparkSession.Read()
    .Format("csv")
    .Options(new Dictionary<string, string>
            {
                { "sep", "|" }
                { "inferSchema", "true" }
            }

The second way to specify the schema is by using the Schema method. The Schema method takes a Data Definition Language string as a parameter.

After specifying the format, the delimiter, and the schema, we set the path to the file by using the Load method to load input in as DataFrame. The Load method is for data sources that require a path (e.g., data backed by a local or distributed file system).

        static void Main(string[] args)
        {
            var sparkSession = SparkSession.
            Builder().
            GetOrCreate();
            var options = new Dictionary<string, string> {
              {"inferSchema", "false" },
              {"delimiter","|" }
            };

            var schemaString = "inv_date_sk INT," +
                "inv_item_sk INT," +
                "inv_warehouse_sk INT," +
                "inv_quantity_on_hand STRING";

            var csvFile = sparkSession.Read()
            .Format("csv")
            .Options(options)
            .Schema(schemaString)
            .Load(@"path-to-file") ...

Load data using the Csv() method

The DataFrameReader returned by the Read method also provides a Csv method to load files with Comma Separated Value directly. The Csv method expects files with Comma separates fields.

sparkSession.Read()
            .Schema(...)
            .Options(...)
            .Csv(@"file.csv") ...

The load() and format() method are generic functions, and they give the user more control over the loading process whereas the Csv() is a more convenient way to load comma-separated values files directly. Watch the video below to see this in action.

.NET for Apache Spark - Getting Started

Sat, 01 Jun 2019 00:00:00 GMT

Apache Spark is an open source analytics engine for big data and machine learning.
Application for Spark can be written in Scala, Python, R, SQL and now also for the .NET Core Framework using C# and F#.
In this video, I will show you how to install Apache Spark on Windows and write an application using .NET Core and C# to test the installation

SQL Server - Full text search with the Contains predicate

Wed, 06 Mar 2019 00:00:00 GMT

By the end of this video, you will be able to write queries to perform the following types of full-text searches:
1. Simple Term: find specific words 05:17
2. Prefix Term: query for words or phrases starting with a specific text 24:33
3. Generation Term: search for multiple forms of a specific word17:55
4. Synonymous Term: find words with similar meaning 28:58

SQL Server - Introduction to full-text search

Fri, 01 Feb 2019 22:12:03 GMT

Searching is the most fundamental thing we do with a database, most of the time we know the shape of what we are looking for, i.e an ID or specific text, but what if you can’t state exactly what you are looking for?

By the end of this post you will be able to

Explain what is full-text search and when to use it.
Set up a full-text search using T-SQL.
Write queries to match words and phrases.
Write queries to match the meaning but not the exact wording.
Compare and contrast full-text search with the operator LIKE.

What is full-text search?

Full-text search lets users and applications run full-text queries against character-based data, that is data of any of the following types: char, varchar, nchar, nvarchar, text, ntext, image, xml, varbinary(max) and FILESTREAM.

Full-text search provides the capabilities to identifies natural language terms (words or phrases) that satisfy a query, and optionally to sort them by relevance to the query.

When we use full-text search to look for words, we are either looking for an exact match, a less precise match or we are looking for words with similar meaning, this is also known as linguistic search.
With full-text search you can write a query to find a:

Simple Term: specific words or phrases
Prefix Term: words or phrases starting with a specific text
Generation Term: multiple forms of a specific word
Proximity Term: words or phrases close to another words or phrases
Synonymous Term: similar words or phrases

Now you may be thinking huuuh, can’t I do the same using the operator LIKE? The Answer is NO and the reason is that the operator LIKE :

Works only on character pattern
Can’t be used to query binary data like images or documents
Don’t have multiple language support
Can’t do Synonym matching, or fuzzy matching
Can’t do a ranking based on relevance
Can’t do something I can’t think of right now.

Setup

I will use the famous AdventureWorks Database in this Post. If you don’t have it installed, please follow the instructions here.

Full-text search is enabled by default on SQL-Server. There are three basic steps to set up full-text search:

Create a full-text catalog:

The catalog is a logical concept that refers to a group of full-text indexes. One full-text catalog can have several full-text indexes, but a full-text index can only be part of one full-text catalog. Each database can contain zero or more full-text catalogs
Create a unique index on the tables or views of interest
Create a full-text index on the columns of the tables or indexed views of interest

Let’s set up a full-text search on the column Description of the table Production.ProductDescription.

-- Step One: Create a full-text catalog

CREATE FULLTEXT CATALOG
AW2016_FTCatalog
--The catalog name and the names of the files associated with it must
--be unique among all catalog names in the current database
WITH ACCENT_SENSITIVITY = OFF
GO

--Step Two: Create a unique index

CREATE UNIQUE INDEX ui_ukProductDesc
ON Production.ProductDescription(ProductDescriptionID);
--A unique index is one in which no two rows are permitted
--to have the same index key value
--so a primary key column is often a good candidate
GO

-- Step Three: Create a full-text index

CREATE FULLTEXT INDEX ON Production.ProductDescription
(
    Description
    --Full-text index column name that should be indexed
    --and used for full-text search
)
KEY INDEX ui_ukProductDesc ON AW2016_FTCatalog
WITH CHANGE_TRACKING AUTO   --Population type;
GO

What is accent sensitivity ?

A common name in Germany is Müller and it is often stored as Mueller in databases, and if you are querying for Müller you want Mueller to be returned as well, or let’s say in french the word cafe and café should also interchangeable.

Accent sensitivity is related to whether a diacritical mark is taken into consideration when comparing two strings, by default the accent-sensitivity specified in the database collation is used.

Use the following query to get accent sensitivity of a catalog

select FULLTEXTCATALOGPROPERTY('catalogName','AccentSensitivity')

to get the accent sensitivity of a database, get the collation of the database and check for the last two character, for example SQL_Latin1_General_CP1_CI_AS is accent sensitiv (AS) and SQL_Latin1_General_CP1_CI_AI is not

To view the collation setting of a database:

SELECT name,
       collation_name
FROM sys.databases
WHERE name=N'DatabaseName';
GO

The Querying

T-SQL offer two predicates and two functions for full-text search

CONTAINS and CONTAINSTABLE to match words and phrases
FREETEXT and FREETEXTTABLE to match the meaning, but not the exact wording

CONTAINS

The CONTAINS predicate can be used in the WHERE clause of a select statement

We can use the following query to look for all product containing the word Mountain in their description

SELECT p.ProductID,
       p.Name,
       p.Size,
       pd.Description,
       pmpdc.CultureID
FROM Production.Product p
INNER JOIN
Production.ProductModel pm
ON pm.ProductModelID=p.ProductModelID
    INNER JOIN
    Production.ProductModelProductDescriptionCulture pmpdc
    ON pmpdc.ProductModelID=p.ProductModelID
        INNER JOIN
        Production.ProductDescription pd
        ON pd.ProductDescriptionID=pmpdc.ProductDescriptionID
WHERE CONTAINS(Description, 'Mountain');

what’s really cool here, is the fact that the result set also contains product with the word Mountain in other languages than English like Arabic (CultureID: ar) or traditional Chinese (CultureID: Cuzh-cht).

How would you get the same result using the LIKE operator? better go and sign up for a bunch of language courses.

to search for a phrase instead of a single word, put the phrase in double quotes:

.... WHERE CONTAINS(Description, '"Mountain bike"')

if you want to sort the result based on their relevance to the search term, use the CONTAINSTABLE function

CONTAINSTABLE

suppose we are looking for a description that contains the words mountain, bike or wheel, to achieve this, we can use proximity term NEAR to search for words or phrases near one another. CONTAINSTABLE returns a column named RANK containing values from 1 to 1000. These values are used to rank the rows returned according to how well they match the selection criteria

SELECT p.ProductID,
       p.Name,
       p.Size,
       pd.Description,
       pmpdc.CultureID,
       KEY_TBL.RANK
FROM Production.Product
AS p
INNER JOIN
Production.ProductModel
AS pm
ON pm.ProductModelID=p.ProductModelID
    INNER JOIN
    Production.ProductModelProductDescriptionCulture
AS pmpdc
    ON pmpdc.ProductModelID=p.ProductModelID
        INNER JOIN
        Production.ProductDescription
AS pd
        ON pd.ProductDescriptionID=pmpdc.ProductDescriptionID
            INNER JOIN
            CONTAINSTABLE(Production.ProductDescription,
            Description,
            '(Mountain NEAR bike) OR (wheel NEAR Mountain)')
AS KEY_TBL
            ON pd.ProductDescriptionID=KEY_TBL.[KEY]
    ORDER BY KEY_TBL.RANK DESC;

FREETEXT

To match single words and phrases with precise, less or fuzzy precision we use CONTAINS or CONTAINSTABLE. To match the meaning, but not the exact wording, of specified words or phrases we use FREETEXT or FREETEXTTABLE

if we use the CONTAINS predicate the search for term Mountain bike the result set contains only one row

SELECT ProductDescriptionID,
       Description
FROM Production.ProductDescription
WHERE CONTAINS(Description, '"mountain bike"');

but the FREETEXT predicate returns more than one row

SELECT ProductDescriptionID,
       Description
FROM Production.ProductDescription
WHERE FREETEXT(Description, 'mountain bike');

This is because FREETEXT also returns rows that contain the word bike or mountain.

FREETEXTTABLE

The FREETEXTTABLE function works pretty similarly to the FREETEXT predicate except for the fact, that it returns a table and the result can be ranked by relevance to the search term

let’s use FREETEXTTABLE to look up for product description containing the phrase Mountain bike

SELECT KEY_TBL.RANK,
       FT_TBL.Description,
       FT_TBL.ProductDescriptionID
FROM Production.ProductDescription
AS FT_TBL
INNER JOIN
FREETEXTTABLE(Production.ProductDescription,
              Description, 'mountain bike')
AS KEY_TBL
ON FT_TBL.ProductDescriptionID=KEY_TBL.[KEY]
WHERE KEY_TBL.RANK>=10
ORDER BY KEY_TBL.RANK DESC;
GO

The result of this query consists of all rows containing any variation of the search term. The rows are sorted by their ranking, rows with the best matching have the higher rank and so on.

Summary

Given the fact that today more data are been generated than before, expanding your knowledge about how to effectively find what you’re looking for is an enormous advantage, and if you are building an application that heavily relies on querying character-based data, then the full-text search feature of your RDBMS is worth considering. Full-text search is not only available for SQL-Server but on other RDBMS like PostgreSQL, MySQL and others.

This was just an introduction to full-text search, in the next post we will explore it in more details and take a look at advanced topics like synonym matching, weighted matching or how to use the string functions SOUNDEX and DIFFERENCE to find words based on how they sound when spoken