finaquant.com

Calling table functions from Microsoft Excel

admin — Sun, 26 May 2013 11:34:31 +0000

This article explains how the table functions of the non-commercial .net library finaquant® protos can be called from within Microsoft Excel using VBA macros.

Two possible integration options are presented here:

1) Calling a .net method from an Excel macro
Once called from a macro, the .net method reads the input tables from the given excel worksheets before executing the table functions to obtain output tables. The output tables are in turn written into the desired excel sheets.

Note that names of the worksheets are passed from the macro to the .net method as parameters for selecting the input tables. Each worksheet contains a single input table.

This integration option can be useful for demonstrations that are executed directly from excel. See the code examples below:

Excel VBA macro calling .net method: CalculatePrices()

' Calculate price table with given cost and margin tables
' 1) Read input tables: Costs, Margins, MetaData
' 2) Calculate price table
' 3) Write resultant price table into a sheet
Private Sub CalculatePrices()
Dim pxl As New ProtosExcelFunc
Dim CostTableSheet, MarginTableSheet, MetaDataSheet, PriceTableSheet As String
 
CostTableSheet = "CostTable"
MetaDataSheet = "MetaData"
 
MarginTableSheet = "MarginTable1"
PriceTableSheet = "PriceTable1"
 
'MarginTableSheet = "MarginTable2"
'PriceTableSheet = "PriceTable2"

' call .net method
MsgBox pxl.CalculatePrices(Application.ActiveWorkbook.FullName, CostTableSheet, _
    MarginTableSheet, MetaDataSheet, PriceTableSheet)
End Sub

.net method called by Excel macro: CalculatePrices()

/// 
/// Calculate price table for given cost and margin tables
/// 
/// Full file path of excel file
/// Name of worksheet for reading cost table (input table)
/// Name of worksheet for reading margin table (input table)
/// Name of worksheet for reading field names and types (input table)
/// Name of target worksheet for writing price table (output table)
/// Message to calling function
public string CalculatePrices(string ExcelFilePath, string CostTableSheet, string MarginTableSheet,
	string MetaDataSheet, string PriceTableSheet)
{
	try
	{
		// read meta data
		DataTable MetaTbl = ExcelFunc.ReadTableFromExcelSheet_ODBC(ExcelFilePath, MetaDataSheet);
		MetaData md = MetaData.ReadFieldsFromDataTable(MetaTbl);
 
		// read input tables
		DataTable CostTbl = ExcelFunc.ReadTableFromExcelSheet_ODBC(ExcelFilePath, CostTableSheet);
		DataTable MarginTbl = ExcelFunc.ReadTableFromExcelSheet_ODBC(ExcelFilePath, MarginTableSheet);
 
		// convert DataTable to MatrixTable
		MatrixTable CostTable = MatrixTable.Import_from_DataTable(CostTbl, md);
		MatrixTable MarginTable = MatrixTable.Import_from_DataTable(MarginTbl, md);
 
		// Calculate price table with table function 'MultiplySelectedKeyFigures'
		MatrixTable PriceTable = MatrixTable.MultiplySelectedKeyFigures(CostTable, (MarginTable + 1),
		InputKeyFigTbl1: "costs", InputKeyFigTbl2: "margin", OutputKeyFig: "price",
		JokerMatchesAllvalues: true);
 
		// convert MatrixTable to DataTable
		DataTable PriceTbl = MatrixTable.Export_To_DataTable(PriceTable);
 
		// write price table into excel sheet
		ExcelFunc.WriteTableToExcelSheet(PriceTbl, ExcelFilePath, SheetName: PriceTableSheet, SaveWorkbook: false);
 
		// return affirmative message to caller
		return "Price table is calculated successfully!";
	}
	catch (Exception ex)
	{
		return "CalculatePrices: An error has occured! " + ex.Message;
	}
}

2) Directly executing the .net method
… which reads input tables from the given excel file before making the required table calculations.

In this option, an excel sheets are used solely as a database to store input and output tables; no additional VBA code (Excel macro) is needed. See the code example below:

.net code for calculating prices: CalculatePriceTable()

/// 
/// Calculate price table with given cost and margin tables
/// 
public static void CalculatePriceTable()
{
	// Get full file path
	string filepath = System.IO.Path.GetFullPath(@"Resources\DotNetIntegration.xlsm");
 
	// call table function
	ProtosExcelFunc pxl = new ProtosExcelFunc();
	string msg = pxl.CalculatePrices(filepath, CostTableSheet: "CostTable", MarginTableSheet: "MarginTable1",
		MetaDataSheet: "MetaData", PriceTableSheet: "PriceTable1");
 
	System.Diagnostics.Debug.WriteLine("Message from CalculatePrices: " + msg);
}

All the example codes above and others can be found in MS Visual Studio File ProtosExcelIntegration 1.04.

Calling a .net method from an Excel macro

Install the latest release of the non-commercial .net library finaquant® protos on your computer.
Ensure that the proper ODBC driver is installed on your computer. See related article for more information.
Download and unzip the latest release of MS Visual Studio File ProtosExcelIntegration 1.04
Open the VS file ProtosExcelIntegration as administrator: Find the application (Visual Studio) in program starter menu, right mouse-click and select Run as administrator.

Ensure that Microsoft Office and Excel Object libraries are referenced by the VS project. You need to update references if you have a Microsoft Office version other than 2010.

Ensure that following VS project configurations are selected:

Application>Output type: Class Library

Build>Output: Check Register for COM interop

Build (or rebuild) the VS project.

Add library COM reference to Excel: Open the excel file under the folder Resources, go to macro VBA view, add reference named Finaquant Protos and Excel Integration from the menu Tools>References

You can now open the excel file under the folder Resources and execute some macros (by pressing F5) that call .net methods for table functions. The VBA macros can be found in Module1 in developer view:

You can also execute a macro (CalculatePrices2 in Module1) by selecting the input tables (by their worksheet names) and pressing a button in the worksheet named RunMacro:

Resultant price table is written into a new worksheet named ProceTable1:

This integration option is particularly useful for demonstrations: Users can first enter the calculation parameters (like sheet names for selecting the input tables here) and execute their table calculation by just pressing a button.

Directly executing the .net method

Open the VS file ProtosExcelIntegration as normal user; you don’t need to open it as administrator.
Ensure that following VS project configurations are selected:
Build>Output: Uncheck Register for COM interop
Application>Output type: Select Console Application
Uncomment the test method you want to execute in code file Program.cs and press F5

Resultant table is written into a new worksheet named CombTable2:

Importing and Exporting a DataTable from/to an Excel file

tuncalik — Thu, 02 May 2013 14:47:35 +0000

As explained in the related article, a MatrixTable of finaquant can easily be converted to or from a DataTable of the .net framework (ADO.net).

That is, importing or exporting a MatrixTable from/to an excel sheet can be accomplished in two steps:

Import: (1) From Excel to DataTable (2) From DataTable to MatrixTable

Export: (1) From MatrixTable to DataTable (2) From DataTable to Excel

Data integration of .net with excel

Tables and values can be exchanged between .net and excel by using the methods in C# listed below.

The requirements for debugging and running these methods:

1) Microsoft Office and Excel must be installed
2) Following excel-related COM assemblies must be referenced:
Microsoft Office XX.0 Object Library
Microsoft Excel XX.0 Object Library
3) Following namespaces must be included in reference list in the code:
using System.Data;
using Excel = Microsoft.Office.Interop.Excel;

Feel free to use the code wherever you want provided that you leave any copyright information for Finaquant Analytics intact. We would appreciate any comment for possible corrections and improvements in the code.

Import DataTable from Excel Sheet using ODCB driver

This method reads a table in an excel sheet into a DataTable object using the ODBC adapter. It requires Microsoft Access Database Engine as driver software. The first line of the table in excel sheet must contain field names.

C# code for ReadTableFromExcelSheet_ODBC()

/// 
/// Reads a table in an excel sheet into DataTable object using ODBC adapter.
/// First line of table must contain field names.
/// 
/// A valid file path like  @"C:\Users\John\Documents\Inventory.xlsx
/// Name of excel sheet
/// Table
public static DataTable ReadTableFromExcelSheet_ODBC(string ExcelFilePath, string SheetName)
{
	// PARAMETER CHECKS
	if (ExcelFilePath == null || ExcelFilePath == "")
		throw new Exception("ReadTableFromExcelSheet_ODBC: Null or empty string FilePath");
	if (SheetName == null || SheetName == "")
		throw new Exception("ReadTableFromExcelSheet_ODBC: Null or empty string SheetName");
	if (! File.Exists(ExcelFilePath))
		throw new Exception("ReadTableFromExcelSheet_ODBC: Excel file is not found!");
 
	// parameter checks OK, continue...
	string ConnStr = "Driver={Microsoft Excel Driver (*.xls, *.xlsx, *.xlsm, *.xlsb)};DBQ=" + ExcelFilePath + ";";
	string Sql = "SELECT * FROM [" + SheetName + "$]";
 
	try
	{
		var adp = new OdbcDataAdapter(Sql, ConnStr);
 
		// fill datatable
		DataTable dt = new DataTable();
		adp.Fill(dt);
		return dt;
	}
	catch (Exception ex)
	{
		throw new Exception("ReadTableFromExcelSheet_ODBC: " + ex.Message);
	}
}

Import DataTable from Excel Sheet using OLEDB driver

This method reads a table from an excel sheet into a DataTable using the OLEDB driver.

C# code for ReadTableFromExcelSheet_OLEDB

/// 
/// Read DataTable from a sheet of excel file.
/// Header line of the table in in excel sheet must contain field names.
/// Copyrights: Finaquant Analytics - www.finaquant.com
/// 
/// DataTable input
/// A valid file path like  @"C:\Users\John\Documents\Inventory.xlsx
/// Name of excel sheet
public static DataTable ReadTableFromExcelSheet_OLEDB(string ExcelFilePath, string SheetName)
{
	// var connectionString = string.Format("Provider=Microsoft.Jet.OLEDB.4.0; data source={0}; Extended Properties=Excel 8.0;", Filepath);
 
	// PARAMETER CHECKS
	if (ExcelFilePath == null || ExcelFilePath == "")
		throw new Exception("ReadTableFromExcelSheet_OLEDB: Null or empty string FilePath");
	if (SheetName == null || SheetName == "")
		throw new Exception("ReadTableFromExcelSheet_OLEDB: Null or empty string SheetName");
	if (!File.Exists(ExcelFilePath))
		throw new Exception("ReadTableFromExcelSheet_OLEDB: Excel file is not found!");
 
	// parameter checks OK, continue...
	// set extended property for connection string
	string ExtendedProperty;
	if (ExcelFilePath.Substring(ExcelFilePath.LastIndexOf('.')).ToLower() == ".xlsx")
		ExtendedProperty = "Excel 12.0 Xml";
	else if (ExcelFilePath.Substring(ExcelFilePath.LastIndexOf('.')).ToLower() == ".xls")
		ExtendedProperty = "Excel 8.0";
	else if (ExcelFilePath.Substring(ExcelFilePath.LastIndexOf('.')).ToLower() == ".xlsm")
		ExtendedProperty = "Excel 12.0 Macro";
	else if (ExcelFilePath.Substring(ExcelFilePath.LastIndexOf('.')).ToLower() == ".xlsb")
		ExtendedProperty = "Excel 12.0";
	else
		throw new Exception("ReadTableFromExcelSheet_OLEDB: Invalid file type!");
 
	// var connectionString = string.Format("Provider=Microsoft.Jet.OLEDB.4.0; data source={0}; Extended Properties=\"Excel 12.0;HDR=Yes;IMEX=1\"", FilePath);
	var connectionString = string.Format("Provider=Microsoft.ACE.OLEDB.12.0; data source={0}; Extended Properties=\"{1};HDR=Yes;IMEX=1\"", ExcelFilePath, ExtendedProperty);
 
	try
	{
		var adapter = new OleDbDataAdapter("SELECT * FROM [" + SheetName + "$]", connectionString);
 
		var TblOut = new DataTable();
		adapter.FillSchema(TblOut, SchemaType.Source);
		// fill table
		adapter.Fill(TblOut);
		return TblOut;
	}
	catch (Exception ex)
	{
		throw new Exception("ReadTableFromExcelSheet_OLEDB: Could not read table from excel file!\n" + ex.Message + "\n");
	}
}

Import DataTable from a Range in Excel Sheet

This method reads a table from a given range in an excel sheet into a DataTable.

C# code for ReadTableFromExcelRange

/// 
/// Read table from a range in an excel sheet.
/// Header line of the table in in excel sheet must contain field names.
/// Copyrights: Finaquant Analytics - www.finaquant.com
/// 
/// A valid file path like  @"C:\Users\John\Documents\Inventory.xlsx
/// Name of excel sheet
/// A valid range string like "B3:E10"
/// Table
public static DataTable ReadTableFromExcelRange(string ExcelFilePath, string SheetName, string RangeStr)
{
	// PARAMETER CHECKS
	if (ExcelFilePath == null || ExcelFilePath == "")
		throw new Exception("ReadTableFromExcelRange: Null or empty string FilePath");
	if (SheetName == null || SheetName == "")
		throw new Exception("ReadTableFromExcelRange: Null or empty string SheetName");
	if (RangeStr == null || RangeStr == "")
		throw new Exception("ReadTableFromExcelRange: Null or empty string RangeStr");
	if (!File.Exists(ExcelFilePath))
		throw new Exception("ReadTableFromExcelRange: Excel file is not found!");
 
	// parameter checks OK ...
	object misValue = System.Reflection.Missing.Value;
	var xlApp = new Excel.Application();
	bool IfReadOnly = true;
	Excel.Worksheet xlWorkSheet = null;
	Excel.Workbook xlWorkBook = null;
 
	try
	{
		// open workbook
		xlWorkBook = xlApp.Workbooks.Open(ExcelFilePath, 0, IfReadOnly, 5, "", "", true,
				Microsoft.Office.Interop.Excel.XlPlatform.xlWindows, "\t", false, false, 0, true, 1, 0);
 
		// check if sheet exists
		bool SheetExists = false;
 
		foreach (Excel.Worksheet sheet in xlWorkBook.Sheets)
		{
			if (sheet.Name.Equals(SheetName))
			{
				SheetExists = true;
				break;
			}
		}
 
		if (!SheetExists)
			throw new Exception("ReadTableFromExcelRange: Sheet is not found in excel file!");
 
		// get worksheet
		xlWorkSheet = (Excel.Worksheet)xlWorkBook.Sheets.get_Item(SheetName);
 
		// get range object
		Excel.Range myRange = (Excel.Range)xlWorkSheet.get_Range(RangeStr);
 
		// initiate data table
		DataTable tbl = new DataTable();
 
		// get column names from header line
		for (int i = 0; i < myRange.Columns.Count; i++)
		{
			object CellValue = (object) myRange.Cells[1, i + 1].Value2;
			tbl.Columns.Add(CellValue.ToString(), typeof(object));
		}
 
		// read values row by row
		for (int i = 0; i < myRange.Rows.Count - 1; i++)
		{
			tbl.Rows.Add();
 
			for (int j = 0; j < myRange.Columns.Count; j++)
			{
				object CellValue = (object) myRange.Cells[2 + i, j + 1].Value2;
				tbl.Rows[i][j] = CellValue;
			}
		}
		// return data table
		return tbl;
	}
	catch (Exception ex)
	{
		throw new Exception("ReadTableFromExcelRange: " + ex.Message);
	}
	finally
	{
		try
		{
			releaseObject(xlWorkSheet);
			releaseObject(xlWorkBook);
			releaseObject(xlApp);
		}
		catch { }
	}
}
//Helper function for releasing unused resources
public static void releaseObject(object obj)
{
	try
	{
		System.Runtime.InteropServices.Marshal.ReleaseComObject(obj);
		obj = null;
	}
	catch (Exception ex)
	{
		obj = null;
		System.Diagnostics.Debug.WriteLine("Exception Occured while releasing object " + ex.ToString());
	}
	finally
	{
		GC.Collect();
	}
}

Read a Value from a Cell in Excel Sheet

This method reads the value of a cell of an excel sheet.

C# code for ReadValueFromExcelCell

/// 
/// Read value from a cell of an excel sheet.
/// Copyrights: Finaquant Analytics - www.finaquant.com
/// 
/// A valid file path like  @"C:\Users\John\Documents\Inventory.xlsx
/// Name of excel sheet
/// A valid cell string like "B3"
/// Scalar value
public static object ReadValueFromExcelCell(string ExcelFilePath, string SheetName, string CellStr)
{
	// PARAMETER CHECKS
	if (ExcelFilePath == null || ExcelFilePath == "")
		throw new Exception("ReadValueFromExcelCell: Null or empty string FilePath");
	if (SheetName == null || SheetName == "")
		throw new Exception("ReadValueFromExcelCell: Null or empty string SheetName");
	if (CellStr == null || CellStr == "")
		throw new Exception("ReadValueFromExcelCell: Null or empty string CellStr");
	if (!File.Exists(ExcelFilePath))
		throw new Exception("ReadValueFromExcelCell: Excel file is not found!");
 
	// parameter checks OK ...
	object misValue = System.Reflection.Missing.Value;
	var xlApp = new Excel.Application();
	bool IfReadOnly = true;
	Excel.Worksheet xlWorkSheet = null;
	Excel.Workbook xlWorkBook = null;
 
	try
	{
		// open workbook
		xlWorkBook = xlApp.Workbooks.Open(ExcelFilePath, 0, IfReadOnly, 5, "", "", true,
				Microsoft.Office.Interop.Excel.XlPlatform.xlWindows, "\t", false, false, 0, true, 1, 0);
 
		// check if sheet exists
		bool SheetExists = false;
 
		foreach (Excel.Worksheet sheet in xlWorkBook.Sheets)
		{
			if (sheet.Name.Equals(SheetName))
			{
				SheetExists = true;
				break;
			}
		}
 
		if (!SheetExists)
			throw new Exception("ReadValueFromExcelCell: Sheet is not found in excel file!");
 
		// get worksheet
		xlWorkSheet = (Excel.Worksheet)xlWorkBook.Sheets.get_Item(SheetName);
 
		// get range object
		Excel.Range myRange = (Excel.Range)xlWorkSheet.get_Range(CellStr);
 
		// initiate data table
		object CellValue = (object)myRange.Cells[1, 1].Value2;
		return CellValue;
	}
	catch (Exception ex)
	{
		throw new Exception("ReadValueFromExcelCell: " + ex.Message);
	}
	finally
	{
		try
		{
			releaseObject(xlWorkSheet);
			releaseObject(xlWorkBook);
			releaseObject(xlApp);
		}
		catch { }
	}
}

Export DataTable to Excel File

This method writes a DataTable into the first sheet of a new excel file. It makes the excel file open and visible for the user.

C# code for WriteTableToExcelFile

/// 
/// Write table into a new excel file. 
/// Make excel file visible to user after writing.
/// 
/// Table
public static void WriteTableToExcelFile(DataTable tbl)
{
	if (tbl == null || tbl.Columns.Count == 0)
		throw new Exception("WriteTableToExcelFile: Null or empty input table!\n");
	try
	{
		// load excel, and create a new workbook
		Excel.Application excelApp = new Excel.Application();
		excelApp.Workbooks.Add();
 
		// single worksheet
		Excel._Worksheet workSheet = excelApp.ActiveSheet;
 
		// column headings
		for (int i = 0; i < tbl.Columns.Count; i++)
		{
			workSheet.Cells[1, (i + 1)] = tbl.Columns[i].ColumnName;
		}
 
		// rows
		for (int i = 0; i < tbl.Rows.Count; i++)
		{
			// to do: format datetime values before printing
			for (int j = 0; j < tbl.Columns.Count; j++)
			{
				workSheet.Cells[(i + 2), (j + 1)] = tbl.Rows[i][j];
			}
		}
 
		// make excel visible for user
		excelApp.Visible = true;
	}
	catch (Exception ex)
	{
		throw new Exception("WriteTableToExcelFile: " + ex.Message);
	}
}

Export DataTable to Excel Sheet

This method writes a DataTable into an excel sheet. A new sheet is created if the sheet with the given name does not exist.

C# code for WriteTableToExcelSheet

/// 
/// Write table to excel sheet.
/// Copyrights: Finaquant Analytics - www.finaquant.com
/// 
/// Table
/// File path for excel file. An error is raised if file is not found.
/// Name of sheet. A new sheet with given name is created if it does not exist.
/// A cell address like "B3". Upper left corner for the table to be inserted.
/// If true, clear all sheet content before writing table.
/// If true, save workbook without making it visible (open) to user.
public static void WriteTableToExcelSheet(DataTable tbl, string ExcelFilePath, string SheetName,
	string CellStr = "A1", bool ClearSheetContent = true, bool SaveWorkbook = true)
{
	// PARAMETER CHECKS
	if (ExcelFilePath == null || ExcelFilePath == "")
		throw new Exception("WriteTableToExcelSheet: Null or empty string FilePath");
	if (SheetName == null || SheetName == "")
		throw new Exception("WriteTableToExcelSheet: Null or empty string SheetName");
	if (!File.Exists(ExcelFilePath))
		throw new Exception("WriteTableToExcelSheet: Excel file is not found!");
 
	// parameter checks OK ...
	object misValue = System.Reflection.Missing.Value;
	var xlApp = new Excel.Application();
	bool IfReadOnly = false;
	Excel.Worksheet xlWorkSheet = null;
	Excel.Workbook xlWorkBook = null;
 
	try
	{
		// open workbook
		xlWorkBook = xlApp.Workbooks.Open(ExcelFilePath, 0, IfReadOnly, 5, "", "", true,
				Microsoft.Office.Interop.Excel.XlPlatform.xlWindows, "\t", false, false, 0, true, 1, 0);
 
		// check if sheet exists
		bool SheetExists = false;
 
		foreach (Excel.Worksheet sheet in xlWorkBook.Sheets)
		{
			if (sheet.Name.Equals(SheetName))
			{
				SheetExists = true;
				break;
			}
		}
 
		// create a new sheet if it doesnot exist
		if (SheetExists)
			xlWorkSheet = (Excel.Worksheet)xlWorkBook.Sheets.get_Item(SheetName);
		else
		{
			xlWorkSheet = (Excel.Worksheet)xlWorkBook.Sheets.Add();
			xlWorkSheet.Name = SheetName;
		}
 
		if (ClearSheetContent)
			xlWorkSheet.Cells.ClearContents();  // clear sheet content
 
		Excel.Range myCell = (Excel.Range)xlWorkSheet.get_Range(CellStr);
 
		// write column names as table header (first line)
		for (int i = 0; i < tbl.Columns.Count; i++)
		{
			myCell.Cells[1, i + 1] = tbl.Columns[i].ColumnName;
		}
 
		// write table entries
		DateTime dt;
		for (int i = 0; i < tbl.Rows.Count; i++)
		{
			// to do: format datetime values before printing
			for (int j = 0; j < tbl.Columns.Count; j++)
			{
				if (tbl.Columns[j].DataType == typeof(DateTime))
				{
					dt = (System.DateTime)tbl.Rows[i][j];
					myCell.Cells[i + 2, j + 1] = dt.ToShortDateString();
				}
				else
				{
					myCell.Cells[i + 2, j + 1] = tbl.Rows[i][j].ToString();
				}
			}
		}
 
		if (SaveWorkbook)
		{
			xlWorkBook.Save();
			xlWorkBook.Close(true, misValue, misValue);
			xlApp.Quit();
		}
		else
		{
			xlWorkSheet.Activate();
			xlApp.Visible = true;
		}
	}
	catch (Exception ex)
	{
		throw new Exception("WriteTableToExcelSheet: " + ex.Message);
	}
	finally
	{
		try
		{
			releaseObject(xlWorkSheet);
			releaseObject(xlWorkBook);
			releaseObject(xlApp);
		}
		catch { }
	}
}

Write a Value into a Cell of Excel Sheet

This methods writes a given value into a cell of excel sheet.

C# code for WriteValueToExcelCell

/// 
/// Write a value into a cell of excel file.
/// Copyrights: Finaquant Analytics - www.finaquant.com
/// 
/// Value
/// File path for excel file. An error is raised if file is not found.
/// Name of sheet. A new sheet with given name is created if it does not exist.
/// A cell address like "B3" for the value to be inserted.
/// If true, clear all sheet content before writing table.
/// If true, save workbook without making it visible (open) to user.
public static void WriteValueToExcelCell(object CellValue, string ExcelFilePath, string SheetName,
	string CellStr, bool ClearSheetContent = false, bool SaveWorkbook = true)
{
	// PARAMETER CHECKS
	if (CellValue == null)
		throw new Exception("WriteValueToExcelCell: Null-valued cell value");
	if (ExcelFilePath == null || ExcelFilePath == "")
		throw new Exception("WriteValueToExcelCell: Null or empty string FilePath");
	if (SheetName == null || SheetName == "")
		throw new Exception("WriteValueToExcelCell: Null or empty string SheetName");
	if (!File.Exists(ExcelFilePath))
		throw new Exception("WriteValueToExcelCell: Excel file is not found!");
 
	// parameter checks OK ...
	object misValue = System.Reflection.Missing.Value;
	var xlApp = new Excel.Application();
	bool IfReadOnly = false;
	Excel.Worksheet xlWorkSheet = null;
	Excel.Workbook xlWorkBook = null;
 
	try
	{
		// open workbook
		xlWorkBook = xlApp.Workbooks.Open(ExcelFilePath, 0, IfReadOnly, 5, "", "", true,
				Microsoft.Office.Interop.Excel.XlPlatform.xlWindows, "\t", false, false, 0, true, 1, 0);
 
		// check if sheet exists
		bool SheetExists = false;
 
		foreach (Excel.Worksheet sheet in xlWorkBook.Sheets)
		{
			if (sheet.Name.Equals(SheetName))
			{
				SheetExists = true;
				break;
			}
		}
 
		// create a new sheet if it doesnot exist
		if (SheetExists)
			xlWorkSheet = (Excel.Worksheet)xlWorkBook.Sheets.get_Item(SheetName);
		else
		{
			xlWorkSheet = (Excel.Worksheet)xlWorkBook.Sheets.Add();
			xlWorkSheet.Name = SheetName;
		}
 
		if (ClearSheetContent)
			xlWorkSheet.Cells.ClearContents();  // clear sheet content
 
		Excel.Range myCell = (Excel.Range)xlWorkSheet.get_Range(CellStr);
 
		myCell.Cells[1, 1] = CellValue;
 
		if (SaveWorkbook)
		{
			xlWorkBook.Save();
			xlWorkBook.Close(true, misValue, misValue);
			xlApp.Quit();
		}
		else
		{
			xlWorkSheet.Activate();
			xlApp.Visible = true;
		}
	}
	catch (Exception ex)
	{
		throw new Exception("WriteValueToExcelCell: " + ex.Message);
	}
	finally
	{
		try
		{
			releaseObject(xlWorkSheet);
			releaseObject(xlWorkBook);
			releaseObject(xlApp);
		}
		catch { }
	}
}

References and useful links

Converting a MatrixTable to a DataTable, and vice versa

admin — Wed, 01 May 2013 21:10:57 +0000

MatrixTable is a type (class) for in-memory representation of data tables just like DataTable of the .net framework (ADO.net). MatrixTable has but a much simplified data structure compared to DataTable with primary focus on mathematical (or analytical) table functions rather than typical sql-based data operations like inserts and updates.

With the following methods of the MatrixTable class, a DataTable can easily be converted into a MatrixTable, and vice versa:

// From DataTable to MatrixTable
public static MatrixTable Import_from_DataTable(DataTable dtable, MetaData md,
   string TextReplaceNull = "NULL", int NumReplaceNull = 0, double KeyFigReplaceNull = 0)
 
// From MatrixTable to DataTable
public static DataTable Export_To_DataTable(MatrixTable tbl)

You can also find open-source DataTable extensions (c#/.net, license: GPL) that include methods for these conversions at CodePlex.

Converting DataTable to MatrixTable

In order to convert a DataTable object into a MatrixTable all the fields (columns) of DataTable must be predefined in MetaData with proper field types.

See the c# code example below:

using FinaquantProtos;
using System.Data;
 
// First, create a DataTable object
// For a helpful reference see: http://www.dotnetperls.com/datatable
DataTable DTbl = new DataTable();
DTbl.Columns.Add("product", typeof(string));
DTbl.Columns.Add("model", typeof(int));
DTbl.Columns.Add("date", typeof(DateTime));
DTbl.Columns.Add("sales", typeof(double));
 
// Add some rows to DataTable
DTbl.Rows.Add("Computer", 2010, DateTime.Now, 25000.0);
DTbl.Rows.Add("Telephone", 2008, new DateTime(2010, 8, 30), 8000.0);
DTbl.Rows.Add("Camera", 2012, new DateTime(2012, 9, 1), 34000.0);
 
// define all fields of DataTable in meta data
MetaData md = MetaData.CreateEmptyMetaData();
MetaData.AddNewField(md, "product", FieldType.TextAttribute);
MetaData.AddNewField(md, "model", FieldType.IntegerAttribute);
MetaData.AddNewField(md, "date", FieldType.DateAttribute);
MetaData.AddNewField(md, "sales", FieldType.KeyFigure);
 
// convert DataTable to MatrixTable
MatrixTable MTbl = MatrixTable.Import_from_DataTable(DTbl, md);
 
// view MatrixTable
MatrixTable.View_MatrixTable(MTbl, "MatrixTable converted from DataTable");

Converting MatrixTable to DataTable

Quite straightforward, see the code example below:

// make some changes in MatrixTable
MTbl.SetFieldValue(FieldName: "product", RowInd: 0, FieldValue: "Laptop");
MTbl.SetFieldValue(FieldName: "model", RowInd: 1, FieldValue: 1999);
 
// convert MatrixTable to DataTable
DTbl = MatrixTable.Export_To_DataTable(MTbl);
 
// view DataTable
DTbl.FQ_ViewTable("DataTable converted from MatrixTable");

Persistent Table Array

tuncalik — Tue, 09 Apr 2013 14:00:28 +0000

PersistentTableArray and DataStore are new classes of the non-commercial .net library finaquant® protos (starting from release 1.04) that enable connection to relational databases like MySQL and Microsoft SQL in order to read/write complete tables from/to a database together with their instance information.

Each table instance is an element of a persistent table array. Following C# code example illustrates how separate cost tables for each country and year can be stored in a MySQL database with a persistent table array:

using FinaquantProtos;
using System.Data.Odbc;
 
// Connection string for MySQL database: 
// Replace it with your own valid connection string on your computer
string ConnStr = @"Driver={MySQL ODBC 5.2w Driver};Server=127.0.0.1;Database=testdb;uid=root;pwd=";
 
// Data provider
// You need to install ODBC driver for MySQL on your computer (win 32 bit version)
// see: http://dev.mysql.com/downloads/connector/odbc/
string DataProvider = "System.Data.Odbc";
 
// initiate DataStore object for database connection
DataStore datstore = new DataStore(Provider: DataProvider); 
 
// open database connection
datstore.OpenConnection(ConnStr);
 
// initiate "persistent table array" objects
PersistentTableArray PTblCost = new PersistentTableArray(TableName: "costs", dstore: datstore);
 
// Create TableRow objects as instance keys for tables
TableRow trow1 = TableRow.CreateTableRowWithElements(InstanceRowFields, "Peru", 2008);
TableRow trow2 = TableRow.CreateTableRowWithElements(InstanceRowFields, "Peru", 2012);
TableRow trow3 = TableRow.CreateTableRowWithElements(InstanceRowFields, "Bolivia", 2010);
 
// store each table instance separately into database (yes, just by assigning!)
PTblCost[trow1] = CostTblPeru2008;      // store CostTblPeru2008 with instance information in trow1
PTblCost[trow2] = CostTblPeru2012;
PTblCost[trow3] = CostTblBolivia2010;

Note that CostTblPeru2008, CostTblPeru2012 and CostTblBolivia2010 are different instances of in-memory cost tables (of type MatrixTable) with the same field structure.

All you need to do is prepare an empty database and assign the connection string in order to initiate the DataStore object. All necessary tables are automatically created and updated by PersistentTableArray or DataStore operations.

The statement PTblCost[trow1] reads a complete table with instance parameter trow1 from the database, and returns an in-memory table of type MatrixTable:

MatrixTable CostTbl = PTblCost[trow1];

You can view an element (instance) of a persistent table array PTblCost as follows:

MatrixTable.View_MatrixTable(PTblCost[trow1], "Cost Table, Peru - 2008");

Note that each attribute pair (country and year) identifies a single table instance. In this example we have only two attributes, but a table row as instance key of a persistent table array can have as many attributes as you need.

Following statement writes the in-memory table CostTbl into database together with the instance parameter trow1:

PTblCost[trow1] = CostTbl;

If a table with the same field structure and instance parameter already exists in the database it will be overwritten (unless it is locked, see DataStore methods) just like in ordinary arrays.

Persistent Table Array and DataStore

Methods of DataStore class are responsible for all database operations for reading or writing complete tables.

A persistent table array assignment like PTblCost[trow1] = CostTblPeru2008 triggers following method of the DataStore class:

PTblCost.dataStore.WriteTableInstance(CostTblPeru2008, 
    PTblCost.TableName, trow1);

The WriteTableInstance() method of DataStore above writes the complete table CostTblPeru2008 into the database together with the instance information trow1 handling all required database operations behind the scenes.

Persistent Table Array takes this level of abstraction and simplification one step further: All database read or write operations are executed automatically during the ordinary use of table arrays.

Why are table arrays needed?

Table arrays are useful in cases in which we need to make calculations with certain set of input tables to obtain the desired set of output tables.

As an example, consider the price calculation below with in-memory tables:

using FinaquantProtos;
 
PTblPrice =
    MatrixTable.MultiplySelectedKeyFigures(PTblCost, PTblMargin + 1,
    InputKeyFigTbl1: "costs", InputKeyFigTbl2: "margin", OutputKeyFig: "price");

Assume you need to calculate product prices for every year with different margin and cost tables. That is, the field structure of margin and cost tables remain the same, but their entries and values may vary from year to year.

Let’s add another dimension to the equation: You have different margin and cost tables for each year and country.

Following code example demonstrates how you can calculate and store price tables for each country and year. You can find this demo function and more in the MS Visual Studio project file FinaquantProtosStarter (see page finaquant® protos for free download).

Demo function: Calculate price tables for each country and year

// Calculate price tables for each year and country with Persistent Table Arrays
using FinaquantProtos;
 
public static void Persistent_Table_Array()
{
// required variables
MetaData md;
MatrixTable CostTable, MarginTable1, MarginTable2;
 
// call helper to create test tables
Create_Cost_and_Margin_tables(out md, out CostTable, out MarginTable1, out MarginTable2);
 
// Create separate tables for each country and year
MatrixTable CostTblPeru2008 = CostTable + 0.08;     // add 0.08 to all key figures of CostTable
MatrixTable MarginTblPeru2008 = MarginTable1 + 0.0008;
 
MatrixTable CostTblPeru2012 = CostTable + 0.12;
MatrixTable MarginTblPeru2012 = MarginTable1 + 0.0012;
 
MatrixTable CostTblBolivia2010 = CostTable + 0.10;
MatrixTable MarginTblBolivia2010 = MarginTable1 + 0.0010; 
 
// view tables
MatrixTable.View_MatrixTable(CostTblPeru2008, "CostTblPeru2008");
MatrixTable.View_MatrixTable(MarginTblPeru2008, "MarginTblPeru2008");
 
// add new fields to MetaData
MetaData.AddNewField(md, "country", FieldType.TextAttribute);
MetaData.AddNewField(md, "year", FieldType.IntegerAttribute);
 
// define row field structure as key for table instance
// Note: TableInstance = PersistentTableArray[InstanceKey]
var RowFields = TextVector.CreateVectorWithElements("country", "year");
TableFields InstanceRowFields = TableFields.CreateTableFields(RowFields, md);
 
// Database connection strings: replace them with your own connection strings on your computer
// You need to install ODBC driver for MySQL on your computer (32 bit version)
// see: http://dev.mysql.com/downloads/connector/odbc/ - mysql-connector-odbc-5.2.4-win32
 
// MS SQL
string ConnStr1 = @"Data Source=(local)\SQLEXPRESS;Initial Catalog=finaquantdb;Integrated Security=True;Connect Timeout=15;Encrypt=False;TrustServerCertificate=False";
 
// MySQL
string ConnStr2 = @"Driver={MySQL ODBC 5.2w Driver};Server=127.0.0.1;Database=finaquantdb;uid=root;pwd=";
 
string[] Connections = { ConnStr1, ConnStr2};
 
// database software for each connection string
string[] DataBaseSW = {"MS SQL", "MySQL"};
 
// Data providers for each connection string
string[] DataProviders = { "System.Data.SqlClient", "System.Data.Odbc"};
 
//****************************************************************
// Calculate product prices for each country and year
//   and store each table instance separately
//
// Repeat operations for each connection string
//****************************************************************
 
Console.WriteLine("Calculate product prices for each country and year");
 
// Repeat same operations for each connection string
for (int i = 0; i < Connections.Length; i++)
{
	Console.WriteLine("Database Software: " + DataBaseSW[i] + " *******************");
	Console.WriteLine("Database Provider: " + DataProviders[i]);
	Console.WriteLine("Connection String: " + Connections[i]);
 
	// TEST
	System.Diagnostics.Debug.WriteLine("Database Software: " + DataBaseSW[i] + " *******************");
 
	// declare persistent table arrays
	PersistentTableArray PTblCost = null;
	PersistentTableArray PTblMargin = null;
	PersistentTableArray PTblPrice = null;
 
	// initiate DataStore for database connection
	DataStore datstore = new DataStore(Provider: DataProviders[i]); 
 
	try
	{
		// open database connection
		datstore.OpenConnection(Connections[i]);
 
		// initiate "persistent table array" objects
		PTblCost = new PersistentTableArray(TableName: "costs", dstore: datstore);
		PTblMargin = new PersistentTableArray(TableName: "margins", dstore: datstore);
		PTblPrice = new PersistentTableArray(TableName: "prices", dstore: datstore);
 
		// Create TableRow objects as instance keys for tables
		TableRow trow1 = TableRow.CreateTableRowWithElements(InstanceRowFields, "Peru", 2008);
		TableRow trow2 = TableRow.CreateTableRowWithElements(InstanceRowFields, "Peru", 2012);
		TableRow trow3 = TableRow.CreateTableRowWithElements(InstanceRowFields, "Bolivia", 2010);
		TableRow trow4 = TableRow.CreateTableRowWithElements(InstanceRowFields, "Bolivia", 2012);
 
		// store each table instance separately into database (yes, just by assigning!)
		PTblCost[trow1] = CostTblPeru2008;      // store CostTblPeru2008 with instance information in trow1
		PTblCost[trow2] = CostTblPeru2012;
		PTblCost[trow3] = CostTblBolivia2010;
 
		PTblMargin[trow1] = MarginTblPeru2008;
		PTblMargin[trow2] = MarginTblPeru2012;
		PTblMargin[trow3] = MarginTblBolivia2010;
 
		// now check database to see if these tables are written
 
		// read table instances from database
		// "PTblCost[trow1]" reads table named "costs" with instance information in trow1
		MatrixTable.View_MatrixTable(PTblCost[trow1], "Cost Table, Peru - 2008");
		MatrixTable.View_MatrixTable(PTblCost[trow2], "Cost Table, Peru - 2012");
		MatrixTable.View_MatrixTable(PTblMargin[trow1], "Margin Table, Peru - 2008");
 
		MatrixTable.View_MatrixTable(PTblCost[trow3], "Cost Table, Bolivia - 2010");
		MatrixTable.View_MatrixTable(PTblMargin[trow3], "Margin Table, Bolivia - 2010");
 
		// calculate and store price tables for each instance (country-year)
		PTblPrice[trow1] =
			MatrixTable.MultiplySelectedKeyFigures(PTblCost[trow1], PTblMargin[trow1] + 1,
			InputKeyFigTbl1: "costs", InputKeyFigTbl2: "margin", OutputKeyFig: "price");
 
		PTblPrice[trow2] =
			MatrixTable.MultiplySelectedKeyFigures(PTblCost[trow2], PTblMargin[trow2] + 1,
			InputKeyFigTbl1: "costs", InputKeyFigTbl2: "margin", OutputKeyFig: "price");
 
		PTblPrice[trow3] =
			MatrixTable.MultiplySelectedKeyFigures(PTblCost[trow3], PTblMargin[trow3] + 1,
			InputKeyFigTbl1: "costs", InputKeyFigTbl2: "margin", OutputKeyFig: "price");
 
		// overwrite a table instance with itself
		PTblPrice[trow3] = PTblPrice[trow3];
 
		// coply a table instance to another one
		PTblPrice[trow4] = PTblPrice[trow3];
 
		// read price tables from database
		MatrixTable.View_MatrixTable(PTblPrice[trow1], "Price Table, Peru - 2008");
		MatrixTable.View_MatrixTable(PTblPrice[trow2], "Price Table, Peru - 2012");
		MatrixTable.View_MatrixTable(PTblPrice[trow3], "Price Table, Bolivia - 2010");
		MatrixTable.View_MatrixTable(PTblPrice[trow4], "Price Table, Bolivia - 2012 (copy of Bolivia-2010)");
	}
	catch (Exception ex)
	{
		Console.WriteLine("Error in Case 1: " + ex.Message);
	}
	finally
	{
		Console.WriteLine("Close database connection ...");
 
		datstore.CloseConnection();
	}
 
	// wait a bit before showing message box
	System.Threading.Thread.Sleep(1000);
	MessageBox.Show("Please close all table viewers and press OK to continue ...");
}   // for i
}

There are lots of similar applications in real life where you need to make instance calculations with selected set of input tables. Examples include financial planning (quarterly, yearly, for each region), dealer commissions (monthly, quarterly, yearly, for each product or contract group), estimations or forecasts and so on. In most cases, the required input data are selected with ad-hoc, customized or parameterized SQL statements.

Persistent Table Array provides a framework for easily reading and writing complete tables associated with a calculation instance (without needing to care for the underlying database operations) as a calculation engine based on table functions.

Note that no method of the classes PersistentTableArray or DataStore will let you manipulate the entries of individual table instances. Table instances are read, written, overwritten or deleted as a whole in compliance with the philosophy of table functions.

Conclusions

Remember how we define a calculation engine based on table functions (video: What is a calculation engine? Comparisons with excel sheets (video)):

Table function: Functions with in-memory tables as input and output parameters

An extensive library of general-purpose table functions (like standard formulas in excel)
User-defined functions and formulas must be possible (like user-defined formulas in excel)
Ability to store multiple instances of tables in a generic database

Up to release 1.03 finaquant® protos already had the features 1 and 2 as listed above. But the last feature 3 was the missing link for a full-featured table calculation engine.

The new classes Persistent Table Array and DataStore with the new release 1.04 completes this missing link.

Persistent Table Array is also the necessary infrastructure of Calculation Nodes and Networks because every worker ant (as calculation node) should be able to read/write tables from/to a database.

Testing Persistent Table Arrays with a MySQL database

admin — Tue, 09 Apr 2013 13:00:50 +0000

MySQL is definitely one of the most popular database systems. It is free and open-source, and comparably easy to install and use.

The all-in-one installation package makes the installation quite simple. This package also includes MySQL workbench for tasks like executing SQL statements and database administration with a nice graphical user interface (GUI).

PersistentTableArray class of the non-commercial .net library finaquant® protos can be connected with a relational database like MySQL (or Microsoft SQL) in order to read or write in-memory tables from/to the database together with their instance information.

You may find some interesting articles in internet if you search for MySQL versus SQL Express.

PersistentTableArray and DataStore classes of finaquant® protos that are responsible for the database connection don’t need many widgets and gadgets of a sophisticated commercial database; they are based on most basic standard (ANSI) SQL statements for reading or writing complete tables. So technically, MySQL database is more than enough for persistent table arrays with the advantage that it doesn’t have the storage space limitations of Microsoft SQL Server Express.

You will also need Microsoft Visual Studio (VS) to run the demo function for persistent table arrays included in the VS project file FinaquantProtosStarter that you can download at the product page of finaquant® protos.

Initiating a persistent table array is about setting the correct database connection string and data provider to a DataStore object:

using FinaquantProtos;
using System.Data.Odbc;
 
// initiate DataStore for database connection
DataStore datstore = new DataStore(Provider: "System.Data.Odbc");
// open database connection
string ConnStr = @"Driver={MySQL ODBC 5.2w Driver};Server=localhost;Database=finaquant;uid=root;pwd=MyPassword";
datstore.OpenConnection(ConnStr);
 
// initiate a persistent table array
var PTblCost = new PersistentTableArray(TableName: "costs", dstore: datstore);

The software ingredients you need for the test

A computer with a Windows operating system and .Net 4.0 framework (client version)
Finaquant Protos .net library (release 1.04 or higher)
Microsoft Visual Studio C# (2010 Express or higher versions)
MySQL database (all-in-one installation) and ODBC Driver for MySQL (32 bit version)
Visual Studio project file FinaquantProtosStarter including the related demo function

Steps

Following steps explain all the preparations required for running the related demo function in FinaquantProtosStarter.

1) Install Microsoft Visual Studio C# 2010 Express(which is free) or a higher version.

For downloads visit: Microsoft Visual Studio

Select “install with a database” if you are given the option.

2) Download and install the latest release of finaquant® protos.

3) Download and unzip the MS Visual Studio file FinaquantProtosStarter 1.04

You need to be registered and logged in at finaquant.com in in order to download this file.

4) Install MySQL all-in-one package (download at: MySQL downloads)

In the installation user interface you can select what components you need. We recommend you to select:

MySQL Server (all files)
Applications (all applications)
Connectors: You need only Connector/ODBC.

IMPORTANT: If you are working on a 64-bit computer you need to download and install 32-bit version of the ODBC connector separately, because even a 64-bit machine may need a 32-bit connector.

Download ODBC Driver for MySQL (32 bit version, mysql-connector-odbc-5.2.4-win32.msi).

5) Using MySQL Workbench create a new connection and a database named finaquant

See detail steps for help

In the main window you can click on Manage Connection to update or create database connections. You don’t need to change here anything; you can set a password to the default user root if you want.

Select your local connection and click on Open Connection to Start Querying to open the SQL Editor window.

Right-click on an existing database (initially test only) and select Create Schema to create the new database named finaquant.

You have now your new database which is all you need for Persistent Table Array.

6) Set the database connection string in the demo function Persistent_Table_Array() in code file Demo.cs and run the method which calculates price tables for given instances of cost and margin tables.

See detail steps for help

A) Find the demo function public static void Persistent_Table_Array() in code file Demo.cs

B) Paste the connection string in the demo function as updated string value for ConnStr2, and save code file Demo.cs

C) Out-comment all other demo functions, un-comment DemoFunctions.Persistent_Table_Array() in the code file Program.cs and press F5 to run the demo function.

This demo function should calculate and display price tables for each instance (country and year) of cost and margin tables.

7) Check the database to see which table are created.

The table named list_of_data_tables has an entry for each instance of data tables.

A single table of the database stores multiple instances of in-memory cost tables.

Testing Persistent Table Array with a Microsoft SQL database

admin — Tue, 09 Apr 2013 12:00:07 +0000

With the PersistentTableArray class of the non-commercial .net library finaquant® protos you can read and write in-memory tables from/to a relational database together with their instance information.

If you have already installed Microsoft Visual Studio C# 2010 Express or a higher version on your computer it is quite easy to get started with persistent table arrays. The installation file of Visual Studio is packed with a free database SQL Server Express you don’t need to install a database software separately.

You will need Visual Studio in any case to run the demo function for persistent table arrays in the Visual Studio project file FinaquantProtosStarter which you can download at the product page of finaquant® protos.

Initiating a persistent table array is about setting the correct database connection string and data provider to a DataStore object:

using FinaquantProtos;
using System.Data.SqlClient;
 
// initiate DataStore for database connection
DataStore datstore = new DataStore(Provider: "System.Data.SqlClient");
 
// open database connection
string ConnStr = “@"Data Source=.\SQLEXPRESS;AttachDbFilename=C:\Users\Tuncali\Documents\finaquant.mdf;Integrated Security=True;Connect Timeout=30;User Instance=True"
datstore.OpenConnection(ConnStr);
 
// initiate a persistent table array
var PTblCost = new PersistentTableArray(TableName: "costs", dstore: datstore);

The software ingredients you need for the test

A computer with a Windows operating system and .Net 4.0 framework (client version)
Finaquant Protos .net library (release 1.04 or higher)
Microsoft Visual Studio C# (2010 Express or higher versions) that comes with an MS SQL Server Express database
Visual Studio project file FinaquantProtosStarter with the related demo function

Steps

Following steps explain all the preparations required for running the related demo function in FinaquantProtosStarter.

1) Install Microsoft Visual Studio C# 2010 Express(which is free) or a higher version.

For downloads visit: Microsoft Visual Studio

Select “install with a database” if you are given the option.

2) Download and install the latest release of finaquant® protos.

3) Download and unzip the MS Visual Studio file FinaquantProtosStarter 1.04

You need to be registered and logged in at finaquant.com in in order to download this file.

4) Open FinaquantProtosStarter solution file (with Visual Studio) and create a new database named finaquant.

See detail steps for help

A) Open FinaquantProtosStarter solution file (with Visual Studio) by double-clicking on it.

B) Select Database Explorer from the menu “View>Other Windows” to create a new database

Persistent Table Array needs an initially empty and dedicated database.

The menu path can be slightly different in 2012 versions of Visual Studio, like:
View>Other Windows>Data Sources, or View>SQL Server Object Explorer

C) Press icon Connect to Database and …

… enter the name of the new database (finaquant) in the opening window. Then press OK without making any other changes.

Confirm the next window asking whether the database should be created with OK.

New database named finaquant is created:

5) Set the database connection string in the demo function Persistent_Table_Array() in code file Demo.cs and run the method which calculates price tables for given instances of cost and margin tables.

See detail steps for help

A) Find the demo function public static void Persistent_Table_Array() in code file Demo.cs

B) In order to get the connection string right-click on the database and select properties.

C) In the properties window right-click on the field value for connection string, and select Select All, press Control-C to store the connection string to clipboard.

D) Paste the connection string in the demo function as updated string value for ConnStr1, and save code file Demo.cs

E) Out-comment all other demo functions, un-comment DemoFunctions.Persistent_Table_Array() in the code file Program.cs and press F5 to run the demo function.

This demo function should calculate and display price tables for each instance (country and year) of cost and margin tables.

6) Check the database to see which table are created.

The table named list_of_data_tables has an entry for each instance of data tables.

Next step: Worker ant ika (calculation node)

tuncalik — Fri, 01 Mar 2013 21:05:22 +0000

Which societies were the first farmers? When and where did they emerge?

Early civilizations in the Fertile Crescent of Middle East or in China about 9000 BC? Or maybe ancient people in Solomon Islands or in the Highlands of New Guinea?

No, social insects like ants and termites were the first farmers on land; leaf-cutter ants began cultivating a fungus species about 50 million years ago!

Video: Coevolution of the ant and fungi

Watch also: Farming Ants Reveal Evolution Secrets

How about the domestication of animals? Which societies were the first breeders and shepherds?

Dogs (originally wolves) were tamed and enslaved about 12’000 years ago. Other herding animals like sheep and cattle were domesticated around 8000 BC. See history of animal domestication

So, these ancient people were the first animal breeders?

Wrong! Farmer ants keep domesticated aphids (small herbivorous insects) for their sugary honeydew since millions of years.

Video: Crazy Ant Farmers – Weird Nature – BBC animals

In fact, with their workers, soldiers, baby sitters, site builders, leaf-cutters and fungus nurses ants were among the first industrial societies. Whereas we humans have education and indoctrination for the definition of roles in the economic life, ants have their genetic imprinting. Both systems work perfectly!

Due to all these wonderful traits ant “ika” (a worker ant named ika or shortly “ant ika”) is the nickname we have chosen for the upcoming feature for our .net libraries finaquant protos and finaquant calcs, namely calculation node.

Our worker ant ika doesn’t live in forests, grasslands or deserts; it lives in the computers of our users.

We can summarize the advantages of ant ika as follows:

Ant ika never complains or rebels; it works and toils tirelessly on the assigned duty for perfection.
Ant ika is never bored of routine work; it never questions its job, or looks for a change.
Ant ika can co-operate with other ants, and build highly efficient superorganisms (calculation networks in our case).
Ant ika have many legs (in fact 3 pairs) that can symbolize arrows for input and output tables.

All these qualities make our worker ant ika an industrious individual of an industrial society. See this: Interesting facts about ants

What is a calculation node? (nickname: ant ika)

A calculation node encapsulates a high-level table function (also called node function) with a number of input and output parameters. Note that all these parameters are tables without an exception.

Once configured properly a calculation node fetches all the required inputs from the database, executes the assigned table function to generate output tables and writes them into the database.

Ant ika has the ability to calculate and store all the table valued parameters of multiple calculation instances. For example you may want to execute a table function for every month and country for calculating some price estimations.

No problem for ant ika! It fetches all the inputs from the databases you want, executes its table function and writes the results to the databases you want, together with the instance information month and country. Quite useful feature if you need periodic or multidimensional table calculations.

With the addition of calculation node as a new feature (presumably until May 2013) finaquant® protos will be more than a library with table functions; it will become a real calculation engine based on table functions.

The whole range of table functions in the library can be used to formulate high-level node functions.

What sort of analytical problems can be solved with calculation nodes?

Well, considering the holistic approach of finaquant libraries to numerical computation with its table valued parameters, the question must rather be asked the other way around: Which problems can’t be solved with calculation nodes?

Possible application areas include estimations and forecasts based on historical data, scenario analysis, simulations and optimizations, fee and commission calculations for provider-dealer networks, strategic financial planning, reporting and so on.

Calculation networks as superorganisms

As already mentioned above, calculation nodes (worker ants) can cooperate to build calculation networks.

The output of a node can be input for another node in a calculation network -which is a sort of function tree with table valued parameters.

Calculation networks will be a feature of our (upcoming) commercial product finaquant® calcs. This is a relatively complex feature that will probably be required for only advanced scenarios.

So much to calculation nodes and networks. If you want to know more about the secret life of ants¹ I strongly recommend you to watch this documentary film:
ANTS – Nature’s Secret Power (Full)
Written by: Tunc Ali Kütükcüoglu

References and recommended readings

Secret life of ants: Phrase inspired by the masterful documentary film The Private Life of Plants presented by David Attenborough

Table data as input to estimation functions in R

tuncalik — Tue, 19 Feb 2013 17:17:37 +0000

As mentioned earlier in related articles, functions written in other languages like R, matlab or ILNumerics can be integrated with table functions using the high-level constructs like function router¹ and subtable transformer².

Following example scenarios demonstrate how different estimation functions written with the popular free and open-source language R can be applied on selected parts (i.e. subtables) of an input table containing some historical data for price estimations.

You can find all the relevant code in R and C# at the download page, including the C# code for generating test data.

Price estimators in R

For the example scenarios below we have three estimation functions in R, all based on Linear Regression (LR) of the global indicators market index and oil price for delivering price predictions.

PriceEstimatorA: Estimated product price based on estimated market index and oil price
PriceEstimatorB: Estimated product price based on estimated market index only
PriceEstimatorC: Estimated product price based on estimated oil price only

R code for estimation functions A, B, C

# help function for displaying variables
display_variable = function(v, vname) { 
print(paste(vname,' ='),quote = FALSE)
print(v,quote = FALSE)
}
 
# Price Estimation Function A
# Estimation with 1st order Linear Regression
# based on historical market index and oil price
#
# Columns of input matrix HistoricalData:
# market index, oil price, product price
#
# MarketIndex: Estimated market index for the next year
# OilPrice: Estimated oil price for the next year
EstimatedProductPriceA = function(HistoricalData, MarketIndex, OilPrice) { 
# historical market index and oil price
RowCount = nrow(HistoricalData)
X_train = rbind(matrix(1,1,RowCount), t(HistoricalData[,1:2]))
 
# hostorical product prices
Y_train = matrix(HistoricalData[,3], RowCount, 1);
 
# optimal coefficient vector to minimize MSE
Bopt = solve(X_train %*% t(X_train)) %*% X_train %*% Y_train
 
# get estimated product price
X_test = matrix(c(1.0, MarketIndex, OilPrice), nrow=1)
Y_test = X_test %*% Bopt
return (Y_test)
}
 
# Price Estimator B
# Estimation with 1st order Linear Regression
# based on historical market index only
#
# Columns of input matrix HistoricalData:
# market index, product price
#
# MarketIndex: Estimated market index for the next year
EstimatedProductPriceB = function(HistoricalData, MarketIndex) { 
# historical market index and oil price
RowCount = nrow(HistoricalData)
X_train = rbind(matrix(1,1,RowCount), t(HistoricalData[,1]))
 
# hostorical product prices
Y_train = matrix(HistoricalData[,2], RowCount, 1);
 
# optimal coefficient vector to minimize MSE
Bopt = solve(X_train %*% t(X_train)) %*% X_train %*% Y_train
 
# get estimated product price
X_test = matrix(c(1.0, MarketIndex), nrow=1)
Y_test = X_test %*% Bopt
return (Y_test)
}
 
# Price Estimator C
# Estimation with 1st order Linear Regression
# based on historical oil price only
#
# Columns of matrix HistoricalData:
# oil price, product price
#
# OilPrice: Estimated oil price for the next year
EstimatedProductPriceC = function(HistoricalData, OilPrice) { 
# historical market index and oil price
RowCount = nrow(HistoricalData)
X_train = rbind(matrix(1,1,RowCount), t(HistoricalData[,1]))
 
# hostorical product prices
Y_train = matrix(HistoricalData[,2], RowCount, 1);
 
# optimal coefficient vector to minimize MSE
Bopt = solve(X_train %*% t(X_train)) %*% X_train %*% Y_train
 
# get estimated product price
X_test = matrix(c(1.0, OilPrice), nrow=1)
Y_test = X_test %*% Bopt
return (Y_test)
}
 
HistoricalData = matrix(c(1.48, 94.38, 2.89, 0.91, 91.68, 1.83, 1.27, 85.61, 2.74), nrow=3, byrow=TRUE)
print('HistoricalData')
print(HistoricalData)
 
MarketIndex = 1.2
OilPrice = 100
 
# method A
EstPrdPrice = EstimatedProductPriceA(HistoricalData, MarketIndex, OilPrice);
display_variable(EstPrdPrice,'Estimated product price with estimator A')
 
# method B
EstPrdPrice = EstimatedProductPriceB(HistoricalData[,c(1,3)], MarketIndex);
display_variable(EstPrdPrice,'Estimated product price with estimator B')
 
# method C
EstPrdPrice = EstimatedProductPriceC(HistoricalData[,c(2,3)], OilPrice);
display_variable(EstPrdPrice,'Estimated product price with estimator C')

Note that the details of the estimator functions (and whether they make sense in reality) are unimportant for the scenarios below. We will focus on the big picture and integration: How can we apply selected price predictors in R on selected subtables of an input table?

You may see the related articles listed below for more information about estimations based on Linear Regression (LR):

Historical data

Following tables with historical data are used as input for price estimations:

PriceTable: Historical prices of primary products per country and product
IndicatorsTable: Historical values of global indicators market index and oil price

Initializing R functions in C#

// initialize R connection
var envPath = Environment.GetEnvironmentVariable("PATH");
var rBinPath = @"C:\Program Files\R\R-2.14.1\bin\x64";  // check this path on your computer
Environment.SetEnvironmentVariable("PATH", envPath + System.IO.Path.PathSeparator + rBinPath);
 
REngine r_engine = REngine.CreateInstance("RDotNet");
r_engine.Initialize();
 
// initialize price estimation functions in R
 
// price estimator A
string funcstr = @"EstProdPriceFuncA <- function(HistData, EstMarketInd, EstOilPrice) { 
		RowCount = nrow(HistData);
		X_train = rbind(matrix(1,1,RowCount), t(HistData[,1:2]));
		Y_train = matrix(HistData[,3], RowCount, 1);
		Bopt = solve(X_train %*% t(X_train)) %*% X_train %*% Y_train;
		X_test = matrix(c(1.0, EstMarketInd, EstOilPrice), nrow=1);
		Y_test = X_test %*% Bopt;
		return (Y_test);
		}";
Function EstProdPriceFuncA = r_engine.Evaluate(funcstr).AsFunction();
 
// price estimator B
funcstr = @"EstProdPriceFuncB <- function(HistData, EstMarketInd) { 
	RowCount = nrow(HistData);
	X_train = rbind(matrix(1,1,RowCount), t(HistData[,1]));
	Y_train = matrix(HistData[,2], RowCount, 1);
	Bopt = solve(X_train %*% t(X_train)) %*% X_train %*% Y_train;
	X_test = matrix(c(1.0, EstMarketInd), nrow=1);
	Y_test = X_test %*% Bopt;
	return (Y_test);
	}";
Function EstProdPriceFuncB = r_engine.Evaluate(funcstr).AsFunction();
 
// price estimator C
funcstr = @"EstProdPriceFuncC <- function(HistData, EstOilPrice) { 
	RowCount = nrow(HistData);
	X_train = rbind(matrix(1,1,RowCount), t(HistData[,1]));
	Y_train = matrix(HistData[,2], RowCount, 1);
	Bopt = solve(X_train %*% t(X_train)) %*% X_train %*% Y_train;
	X_test = matrix(c(1.0, EstOilPrice), nrow=1);
	Y_test = X_test %*% Bopt;
	return (Y_test);
	}";
Function EstProdPriceFuncC = r_engine.Evaluate(funcstr).AsFunction();

Integration of R with table functions in C#

See related article for the installation of R.NET that we use here for the integration of C# and R.

Function routers and subtable transformers expect delegate table functions as input parameters in the following format:

// Generic delegate table function
// InputTbl: Input table
// OtherParameters: Other parameters of any type
public delegate MatrixTable TransformTableFunc_OP(MatrixTable InputTbl, 
	params Object[] OtherParameters);

The initialized REngine and estimation functions are passed to the enveloping table functions (with the signature above) as additional parameters. The R functions are then called within the enveloping table functions. You may find the code examples below for these table functions:

C# code for table function PriceEstimatorA()

using FinaquantProtos;
using RDotNet;
 
// Table function for price estimation, estimator A (LR based on global market index and oil price)
// Input parameters:
// InputTbl: Historical product prices with fields year and price
// OtherParameters[0]: Table of historical indicators with fields year, market_index and oil_price
// OtherParameters[1]: REngine
// OtherParameters[2]: R Function
// OtherParameters[3]: Estimated market index for next year (double)
// OtherParameters[4]: Estimated oil price for next year (double)
public static MatrixTable PriceEstimatorA(MatrixTable InputTbl, params Object[] OtherParameters)
{
	// get all input parameters
	MatrixTable HistoricalProductPrices = InputTbl;
	MatrixTable HistoricalIndicators = (MatrixTable)OtherParameters[0];
	REngine engine = (REngine)OtherParameters[1];
	Function EstProdPriceFuncA = (Function)OtherParameters[2];
	double[] EstMarketInd = {(double) OtherParameters[3]};
	double[] EstOilPrice = {(double) OtherParameters[4]};
 
	// combine tables
	MatrixTable HistoricalDataTbl = MatrixTable.CombineTables(HistoricalIndicators, HistoricalProductPrices);
 
	// get matrix of key figures from table
	KeyMatrix HistData = HistoricalDataTbl.KeyFigValues;
 
	// convert data types of input parameters from C# to R
	NumericMatrix HistDataR = engine.CreateNumericMatrix(HistData.toArray);
	engine.SetSymbol("HistDataR", HistDataR);
 
	NumericVector EstMarketIndR = engine.CreateNumericVector(EstMarketInd);
	engine.SetSymbol("EstMarketIndR", EstMarketIndR);
 
	NumericVector EstOilPriceR = engine.CreateNumericVector(EstOilPrice);
	engine.SetSymbol("EstOilPriceR", EstOilPriceR);
 
	// call function in R from c#
	NumericMatrix EstProductPrice = engine.Evaluate(
	@"EstProductPrice <- EstProdPriceFuncA(HistDataR,EstMarketIndR,EstOilPriceR)").AsNumericMatrix();
 
	// return a single-element table with estimated product price
	double EstProdPrice = EstProductPrice[0, 0];
 
	// return a single-element table with estimated product price
	TableFields tf = new TableFields(InputTbl.metaData);
	tf.AddNewField("product_price");
	MatrixTable ResultTbl = MatrixTable.CreateTableWithElements_A(tf, EstProdPrice);
	return ResultTbl;
}

C# code for table function PriceEstimatorB()

// Table function for price estimation, estimator B (LR based on global market index only)
// Input parameters:
// InputTbl: Historical product prices with fields year and price
// OtherParameters[0]: Table of historical indicators with fields year, market_index and oil_price
// OtherParameters[1]: REngine
// OtherParameters[2]: R Function
// OtherParameters[3]: Estimated market index for next year (double)
// OtherParameters[4]: Estimated oil price for next year (double)
public static MatrixTable PriceEstimatorB(MatrixTable InputTbl, params Object[] OtherParameters)
{
	// get all input parameters
	MatrixTable HistoricalProductPrices = InputTbl;
	MatrixTable HistoricalIndicators = (MatrixTable)OtherParameters[0];
	REngine engine = (REngine)OtherParameters[1];
	Function EstProdPriceFuncB = (Function)OtherParameters[2];
	double[] EstMarketInd = { (double)OtherParameters[3] };
	double[] EstOilPrice = { (double)OtherParameters[4] };
 
	// combine tables
	MatrixTable HistoricalDataTbl = MatrixTable.CombineTables(HistoricalIndicators, HistoricalProductPrices);
 
	// exclude column oil_price from table
	HistoricalDataTbl = MatrixTable.ExcludeColumns(HistoricalDataTbl, 
		TextVector.CreateVectorWithElements("oil_price"));
 
	// get matrix of key figures from table
	KeyMatrix HistData = HistoricalDataTbl.KeyFigValues;
 
	// convert data types of input parameters from C# to R
	NumericMatrix HistDataR = engine.CreateNumericMatrix(HistData.toArray);
	engine.SetSymbol("HistDataR", HistDataR);
 
	NumericVector EstMarketIndR = engine.CreateNumericVector(EstMarketInd);
	engine.SetSymbol("EstMarketIndR", EstMarketIndR);
 
	NumericVector EstOilPriceR = engine.CreateNumericVector(EstOilPrice);
	engine.SetSymbol("EstOilPriceR", EstOilPriceR);
 
	// call function in R from c#
	NumericMatrix EstProductPrice = engine.Evaluate(
	@"EstProductPrice <- EstProdPriceFuncB(HistDataR,EstMarketIndR)").AsNumericMatrix();
 
	// return a single-element table with estimated product price
	double EstProdPrice = EstProductPrice[0, 0];
 
	// return a single-element table with estimated product price
	TableFields tf = new TableFields(InputTbl.metaData);
	tf.AddNewField("product_price");
	MatrixTable ResultTbl = MatrixTable.CreateTableWithElements_A(tf, EstProdPrice);
	return ResultTbl;
}

C# code for table function PriceEstimatorC()

// Table function for price estimation, estimator C (LR based on global oil price only)
// Input parameters:
// InputTbl: Historical product prices with fields year and price
// OtherParameters[0]: Table of historical indicators with fields year, market_index and oil_price
// OtherParameters[1]: REngine
// OtherParameters[2]: R Function
// OtherParameters[3]: Estimated market index for next year (double)
// OtherParameters[4]: Estimated oil price for next year (double)
public static MatrixTable PriceEstimatorC(MatrixTable InputTbl, params Object[] OtherParameters)
{
	// get all input parameters
	MatrixTable HistoricalProductPrices = InputTbl;
	MatrixTable HistoricalIndicators = (MatrixTable)OtherParameters[0];
	REngine engine = (REngine)OtherParameters[1];
	Function EstProdPriceFuncC = (Function)OtherParameters[2];
	double[] EstMarketInd = { (double)OtherParameters[3] };
	double[] EstOilPrice = { (double)OtherParameters[4] };
 
	// combine tables
	MatrixTable HistoricalDataTbl = MatrixTable.CombineTables(HistoricalIndicators, HistoricalProductPrices);
 
	// exclude column market_index from table
	HistoricalDataTbl = MatrixTable.ExcludeColumns(HistoricalDataTbl,
		TextVector.CreateVectorWithElements("market_index"));
 
	// get matrix of key figures from table
	KeyMatrix HistData = HistoricalDataTbl.KeyFigValues;
 
	// convert data types of input parameters from C# to R
	NumericMatrix HistDataR = engine.CreateNumericMatrix(HistData.toArray);
	engine.SetSymbol("HistDataR", HistDataR);
 
	NumericVector EstMarketIndR = engine.CreateNumericVector(EstMarketInd);
	engine.SetSymbol("EstMarketIndR", EstMarketIndR);
 
	NumericVector EstOilPriceR = engine.CreateNumericVector(EstOilPrice);
	engine.SetSymbol("EstOilPriceR", EstOilPriceR);
 
	// call function in R from c#
	NumericMatrix EstProductPrice = engine.Evaluate(
	@"EstProductPrice <- EstProdPriceFuncC(HistDataR,EstOilPriceR)").AsNumericMatrix();
 
	// return a single-element table with estimated product price
	double EstProdPrice = EstProductPrice[0, 0];
 
	// return a single-element table with estimated product price
	TableFields tf = new TableFields(InputTbl.metaData);
	tf.AddNewField("product_price");
	MatrixTable ResultTbl = MatrixTable.CreateTableWithElements_A(tf, EstProdPrice);
	return ResultTbl;
}

Scenario 1: Applying PriceEstimatorA on a single subtable for country-product pair “Utopia” and “bread”

This example is designed rather for testing if the table function PriceEstimatorA() which envelopes the corresponding R function works as expected.

// apply table function on a single subtable for country-product pair "Utopia", "bread"
 
// filter TemperatureTable with condition table to obtain a subtable
var CondTblFields = TableFields.CreateEmptyTableFields(md);
TableFields.AddNewField(CondTblFields, "country");
TableFields.AddNewField(CondTblFields, "product");
 
var CondTbl = MatrixTable.CreateTableWithElements_A(CondTblFields,
	"Utopia", "bread"
	);
var SubTable = MatrixTable.FilterTableA(PriceTable, CondTbl);
 
// exclude fields country and product from subtable
SubTable = MatrixTable.ExcludeColumns(SubTable,
	TextVector.CreateVectorWithElements("country", "product"));
 
// view subtable
// MatrixTable.View_MatrixTable(SubTable, "Subtable of PriceTable (Utopia, bread)");
 
// execute user-defined table function to calculate expected energy consumption (Sedrun, 2012)
MatrixTable EstimatedPriceTbl = PriceEstimatorA(SubTable, GlobalIndTable, 
	r_engine, EstProdPriceFuncA, 1.2, 90.0);
 
// view result table
MatrixTable.View_MatrixTable(EstimatedPriceTbl, "Estimated price for subtable (Utopia, bread)");

Result table:

Yes, the table function PriceEstimatorA() works as expected.

Scenario 2: Applying PriceEstimatorA on every subtable of price table

In this example, the same prediction function is applied for all countries and products.

Applying the same table (or matrix) function on all subtables of an input table… This is what a subtable transformer is created for.

// Apply the same table function on all subtables of Price Table
// subtable transformer
 
// define subtable fields
var SubTblFields = TableFields.CreateEmptyTableFields(md);
TableFields.AddNewField(SubTblFields, "year");
TableFields.AddNewField(SubTblFields, "product_price");
 
double EstMarketIndex = 1.2;
double EstOilPrice = 100.0;
 
// execute subtable transformer
MatrixTable ResultTbl = MatrixTable.TransformSubTables(PriceTable, SubTblFields,
	PriceEstimatorA, GlobalIndTable, r_engine, EstProdPriceFuncA, EstMarketIndex, EstOilPrice);
 
// view result table
MatrixTable.View_MatrixTable(MatrixTable.Round(ResultTbl,2), 
	"Estimator-A applied for all countries and products");

Result table:

Scenario 3: Applying different price estimators to different countries and products

This is where the fabulous function router comes into play.

What we want to do is summarized in the following table:

In the result table we want to see explicitly which estimation function is used to generate each estimation.

// Function Router and R
// extend MetaData
md.AddNewField("estimation_method", FieldType.TextAttribute);
md.AddNewField("country_cell", FieldType.TextAttribute);
md.AddNewField("product_cell", FieldType.TextAttribute);
 
// create condition matrix table
CondTblFields = TableFields.CreateEmptyTableFields(md);
TableFields.AddNewField(CondTblFields, "country_cell");
TableFields.AddNewField(CondTblFields, "product_cell");
 
var CondMatTbl = MatrixTable.CreateTableWithElements_A(CondTblFields,
	"Utopia", "bread, cheese",      // estimator A
	"Utopia", "butter, honig",      // estimator B
	"Ignoria", "ALL",               // estimator C
	"Euphoria", "ALL"               // estimator A
	);
 
// create associated table to see the estimation method in the result table
var AssocTblFields = TableFields.CreateEmptyTableFields(md);
TableFields.AddNewField(AssocTblFields, "estimation_method");
 
var AssocTbl = MatrixTable.CreateTableWithElements_A(AssocTblFields,
	"estimator A",
	"estimator B",
	"estimator C",
	"estimator A"
	);
// array for delegate functions; assign a function for each row of condition matrix table
var MyTableFuncList = new TransformTableFunc_OP[4];
MyTableFuncList[0] = PriceEstimatorA;
MyTableFuncList[1] = PriceEstimatorB;
MyTableFuncList[2] = PriceEstimatorC;
MyTableFuncList[3] = PriceEstimatorA;
 
// array for other parameters
// pass same parameters for all subtables excluding R Function (estimators A, B, C)
EstMarketIndex = 1.2;
EstOilPrice = 100.0;
 
var OtherParametersList = new object[4][];
OtherParametersList[0] = new object[] { GlobalIndTable, r_engine, EstProdPriceFuncA, EstMarketIndex, EstOilPrice };
OtherParametersList[1] = new object[] { GlobalIndTable, r_engine, EstProdPriceFuncB, EstMarketIndex, EstOilPrice };
OtherParametersList[2] = new object[] { GlobalIndTable, r_engine, EstProdPriceFuncC, EstMarketIndex, EstOilPrice };
OtherParametersList[3] = new object[] { GlobalIndTable, r_engine, EstProdPriceFuncA, EstMarketIndex, EstOilPrice };
 
// call function router C (with condition cell table)
ResultTbl = MatrixTable.FunctionRouterC(PriceTable, SubTblFields,
	CondMatTbl, MyTableFuncList, AssocTbl, FirstMatchOnly: true, IgnoreHierarchy: true,
	ErrorIfConditionNotRelevant: false, OtherParametersList: OtherParametersList);
 
// round all key figures to 2 digits after decimal point
ResultTbl = MatrixTable.Round(ResultTbl, 2);
 
// view result table
MatrixTable.View_MatrixTable(ResultTbl, "Result table: Apply different estimators for different subtables");

Result table:

Scenario 4: Applying all price estimators for all countries and products

In this last scenario we want to apply all available estimation functions on all subtables of the price table in order to compare the estimation results. This is again a task for the function router.

// Function Router and R
// Apply all price estimators to all countries and products
 
// create condition table (not condition matrix table with condition cells!)
CondTblFields = TableFields.CreateEmptyTableFields(md);
TableFields.AddNewField(CondTblFields, "country");
 
CondTbl = MatrixTable.CreateTableWithElements_A(CondTblFields,
	"ALL",                          // estimator A
	"ALL",                          // estimator B
	"ALL"                           // estimator C
	);
// create associated table to see the estimation method in the result table
AssocTbl = MatrixTable.CreateTableWithElements_A(AssocTblFields,
	"estimator A",
	"estimator B",
	"estimator C"
	);
// array for delegate functions; assign a function for each row of condition matrix table
MyTableFuncList = new TransformTableFunc_OP[3];
MyTableFuncList[0] = PriceEstimatorA;
MyTableFuncList[1] = PriceEstimatorB;
MyTableFuncList[2] = PriceEstimatorC;
 
// array for other parameters
// pass same parameters for all subtables excluding R Function (estimators A, B, C)
EstMarketIndex = 1.2;
EstOilPrice = 100.0;
 
OtherParametersList = new object[3][];
OtherParametersList[0] = new object[] { GlobalIndTable, r_engine, EstProdPriceFuncA, EstMarketIndex, EstOilPrice };
OtherParametersList[1] = new object[] { GlobalIndTable, r_engine, EstProdPriceFuncB, EstMarketIndex, EstOilPrice };
OtherParametersList[2] = new object[] { GlobalIndTable, r_engine, EstProdPriceFuncC, EstMarketIndex, EstOilPrice };
 
// call function router A (with condition table)
ResultTbl = MatrixTable.FunctionRouterA(PriceTable, SubTblFields,
	CondTbl, MyTableFuncList, AssocTbl, 
	JokerMatchesAllvalues: true, TextJoker: "ALL", NumJoker: 0,
	FirstMatchOnly: false, OtherParametersList: OtherParametersList);
 
// round all key figures to 2 digits after decimal point
ResultTbl = MatrixTable.Round(ResultTbl, 2);
 
// view result table
MatrixTable.View_MatrixTable(ResultTbl, "Result table: Apply all estimators for all subtables");

Result table:

Conclusions

In this article we demonstrated how selected functions in R can be applied on selected parts (subtables) of input tables using flexible constructs like subtable transformer and function router. In that sense, these constructs can be used to integrate R with data tables containing any number of attributes and key figures.

The same (or similar) integration approach can be used for other computing languages like matlab.

A function router applies selected table (or matrix) functions on selected subtables of an input table.
A subtable transformer applies the same table (or matrix) function on every subtable of an input table.

Calling user-defined functions in R from within c#

admin — Sun, 17 Feb 2013 17:52:48 +0000

In order to integrate R with high-level table constructs like function router¹ or subtable transformer² it might be desirable to call user-defined functions in R from within c#.

This can be done quite easily with the help of the open-source codeplex library R.NET; following steps show how.

You may also want to see the related forum topic:
How can a function router call matrix functions written in R?

1) Open an existing or new project in Microsoft Visual Studio

2) Install R.NET (NuGet)

Installation is quite easy:
Visual Studio (2012) > TOOLS > Library Package Manager > Package Manager Console
type: PM> “Install-Package R.NET”

3) Initialize a function in R and call it from C#
See R.NET documentation for data types in R and R.NET.

using RDotNet;
 
// initialize R connection
var envPath = Environment.GetEnvironmentVariable("PATH");
 
// check the version and path on your computer
var rBinPath = @"C:Program FilesRR-2.14.1binx64";
 
Environment.SetEnvironmentVariable("PATH", envPath 
	+ System.IO.Path.PathSeparator + rBinPath);
 
// initialite REngine object
using (REngine engine = REngine.CreateInstance("RDotNet"))
{
	// Initializes settings.
	engine.Initialize();
 
	// define your function in R as multi-line string
	string myfuncstr = @"matrix_mult <- function(a,b){
	c = a %*% b;
	return(c);
	}";
 
	// embed your R function here
	Function matrix_mult = engine.Evaluate(myfuncstr).AsFunction();
 
        // create input matrices
        NumericMatrix a = engine.Evaluate(@"a <- matrix(c(1,4,2,5,3,6), nrow=2)").AsNumericMatrix();
        NumericMatrix b = engine.Evaluate(@"b <- matrix(c(1,4,2,5,3,6), nrow=3)").AsNumericMatrix();
 
	// call your R function from c#
	NumericMatrix c = engine.Evaluate(@"c <- matrix_mult(a,b)").AsNumericMatrix();
 
	// ...
}

A function router applies selected table (or matrix) functions on selected subtables of an input table.
A subtable transformer applies the same table (or matrix) function on every subtable of an input table.

finaquant products

admin — Thu, 14 Feb 2013 23:48:22 +0000

Uncommercial: finaquant® protos (available now)	Commercial: finaquant® calcs (upcoming: July 2013)

Uncommercial calculation engine based on table functions - Non-commercial & free .net library - Table calculations without any sql-based database programming - Analytical table functions with in-memory tables as input and output parameters (like standard formulas in excel) - A framework for user-defined table functions (like user-defined formulas in excel) - Ability to calculate and store all the table valued parameters of multiple calculation instances in a relational database like MS SQL or MySQL – more info & downloads	Commercial calculation engine based on table functions - All the features and functions of the non-commercial finaquant® protos, plus Calculation Nodes and Networks for enhanced calculation automation, performance and modularity - Integration of table and matrix computations for applications like scenario analysis and estimations based on historical data - Can be used for applications like business analytics and intelligence, financial planning, fee and commission calculations, simulations and optimizations with table data, reporting – more info

finaquant.com

Calling table functions from Microsoft Excel

Calling a .net method from an Excel macro

Directly executing the .net method

Importing and Exporting a DataTable from/to an Excel file

Data integration of .net with excel

Import DataTable from Excel Sheet using ODCB driver

Import DataTable from Excel Sheet using OLEDB driver

Import DataTable from a Range in Excel Sheet

Read a Value from a Cell in Excel Sheet

Export DataTable to Excel File

Export DataTable to Excel Sheet

Write a Value into a Cell of Excel Sheet

References and useful links

Converting a MatrixTable to a DataTable, and vice versa

Converting DataTable to MatrixTable

Converting MatrixTable to DataTable

Persistent Table Array

Persistent Table Array and DataStore

Why are table arrays needed?

Conclusions

Related articles

Testing Persistent Table Arrays with a MySQL database

The software ingredients you need for the test

Steps

Testing Persistent Table Array with a Microsoft SQL database

The software ingredients you need for the test

Steps

Next step: Worker ant ika (calculation node)

What is a calculation node? (nickname: ant ika)

Calculation networks as superorganisms

References and recommended readings

Table data as input to estimation functions in R

Price estimators in R

Historical data

Initializing R functions in C#

Integration of R with table functions in C#

Scenario 1: Applying PriceEstimatorA on a single subtable for country-product pair “Utopia” and “bread”

Scenario 2: Applying PriceEstimatorA on every subtable of price table

Scenario 3: Applying different price estimators to different countries and products

Scenario 4: Applying all price estimators for all countries and products

Conclusions

Calling user-defined functions in R from within c#

finaquant products