Building relatively complex data structures in Powershell

Ok, lets continue our discussion about building an Azure asset database. In this short article I’m going to show a simple way of building a dynamic data structure which stores database alongside with its indexes.

lets start from quick reminder of where we are. At the moment we have a variable, containing database of Azure based VMs. And we have a procedure of building indexes, based on hash tables. How can we use it to search the DB in a cmdlet based way. I think that the best way is to use following semantics:

Find-AzureDBObject -Database $DB -IPAddress

where IPAddress parameter is a name of an index. So here we have two problems. First we need to store name of the index together with index itself (add index operation), and second, we need to enumerate indexes we have added (get index operation). In this case we going to get an option to dynamically add indexes to the structure. On the other hand we need this cmdlet to be able to dynamically add parameters to itself depending on the indexes of the DB.

In my opinion simplest way to achieve this is to build a structure of nested hash tables. For instance like this:


it simply contains database itself and links to a set of indexes. And as I mentioned before each index has links to the original database. Each of these structures is basically a hash table. For instance key described as following:

$DB = @{Database = $ht; Index = @{}}

where $ht is the original database, extracted from Azure. Second field will contain the list of indexes, like this

$Index = @{'IPAddress' = $ipAddrIndex; StorageAccount = $storageAcctIndex; DriveFile = $storageFileIndex}

and as a result we build whole structure like this

#create database

$DB = @{Database = $ht; Index = @{'IPAddress' = $ipAddrIndex; StorageAccount = $storageAcctIndex; DriveFile = $storageFileIndex}}

This gives us an ability to add/remove indexes on the fly, use them and build cmdlets with dynamic parameters to easily use these indexes for search.

Next time we will look at how to use this in cmdlets


Another way of looking for objects in Azure

In this article we are going to have a look at number of things in PowerShell. We are going to try dynamic parameters, parallel execution of scriptblocks, building indexes for simple and fast search and much more.

So lets start from the very beginning and describe the problem. The main issue is that sometimes it is required to search VMs not only by their names but also by their internal IP addresses or even by name of their disks. Unfortunately i have no idea if there is any API for doing that out of the box. At this point i decided to build this kind of a tool myself. Goals were defined as follows:

  1. Extract data from Azure as fast as possible, use parallel threads
  2. Avoid multiple lookups in the array of the VMs. Use some kind of index for this
  3. Have an ability to add indexes if necessary. Search commands should adopt dynamically
  4. Use standard, documented REST calls as much as possible
  5. As far as almost all VMs are V1 use classic REST calls

For those who does not want to read whole article code samples are here. Улыбка

Extracting data.

In our environment we have more then ten subscriptions and this number going to grow. Lookup procedure should be able to find an object across all existing subscriptions and automagically find new as they appear.  The only way i saw here is to extract all data from Azure and store it locally in some kind of array. I did some tests trying to extract this data using good old PowerShell Azure cmdlets like Get-AzureVM and found performance very slow. It took ages to extract data for only one subscription. So i decided to try to run this cmdlet in parallel using PSParallel module. Unfortunately this was not a good idea. It looks like some of the cmdlets inside of MS Azure module are not ready to run in parallel. I suspect that problem is with Select-AzureSubscription part of it. Parallel execution was failing at some completely unpredictable places with really strange error messages. After some time of experimenting i decided to leave this path and switch to Azure REST API and extract data myself. Drawback of this is that it will be impossible to pass returned objects to other Azure cmdlets. But at least we going to be able find them very fast. Will see how we can overcome this in future.

So I stated with listing cloud services in parallel using this call.

$sub = Get-AzureSubscription

$h = $sub.SubscriptionId | Invoke-Parallel { Invoke-RestMethod -uri$_/services/hostedservices -Method GET -Headers $headers }

This works pretty fine and pretty fast. After that i run get deployment call to extract VMs data and store it into hashtable with a service name as a key. After this is completed i have a list of all VMs across all of subscriptions available to me.

$ht = [hashtable]::Synchronized(@{})
$h.HostedServices.HostedService | skip-null| invoke-parallel {
#declare function inside of the script block
function xmlToObject {
$h = @{}
$o  | gm -MemberType Property | % {
$prop = $
if ($o."$prop" -is 'System.Xml.XmlElement') {
$h[$prop] = xmlToObject $o."$prop"
else { $h[$prop] = $o."$prop" }
try {
$curr = $_
$subId = ([uri]$curr.Url).Segments[1] -replace "\/",""
$d = Invoke-RestMethod -uri "<a href=""></a>$subId/services/hostedservices/$($curr.ServiceName)/deploymentslots/production" -Method GET -Headers $headers
if ($d) {
$ri1 = $d.Deployment.RoleInstanceList.RoleInstance | % { xmlToObject $_ }
$ro1 = $d.Deployment.RoleList.Role | % { xmlToObject $_  }
$ht[$curr.ServiceName] = $ri1 | % {$c = $_; $r = $c.'RoleName'; $x = $ro1 | where {$_.'RoleName' -eq $r};  $x.remove('RoleName'); [pscustomobject]($_ + $x) }
else { $ht[$curr.ServiceName] = $null }
catch{ $data = 1 }

There are couple of points here. First one is that data returned by the call is in somewhat strange format. Information about VM is stored in two different places; in RoleInstanceList sub tree and in RoleList sub tree. In order to simplify further index creation I’m doing conversion of mentioned XML sub trees to hashtables, join them together and produce pscustomobject. After all this new object gets added to a hashtable to the appropriate cloud service. Second thing is that this conversion being done by xmlToObject function, which in turn should be added to each “thread” explicitly. That is why the scriptblock to run contains definition of the xmlToObject function along with REST method invocation.

In general this scriptblock extracts all deployments from the Cloud Service, extracts machine details from the result, makes objects for each machine and add this object to a hashtable under a Cloud Service Key. As a result we have list of VMs indexed by Cloud Service Name.

Building indexes

At this point we have all the data we need. But if suddenly we need to find something there it may take too much time, especially if we need to do lookup many times. The simplest way to address this is to build additional indexes. For that we can use powershell hashtables. That is pretty easy:

# prepare indexes
$ipAddrIndex = @{}
$storageAcctIndex = @{}
$storageFileIndex = @{}

#build indexes
$ht.Keys | % {
$ht[$_] | % {
if ($_.ipaddress) {$ipAddrIndex[$_.ipaddress] = $ipAddrIndex[$_.ipaddress] + (,$_)}
else {$ipAddrIndex["noip"] = $ipAddrIndex["noip"] + (,$_)}

$ht.Keys | % {
$ht[$_] | % {
if ($_.OSVirtualHardDisk) {$url = [uri]$_.OSVirtualHardDisk.MediaLink; $storageAcctIndex[$] = $storageAcctIndex[$] + (,$_)}
else {$storageAcctIndex["noDisk"] = $storageAcctIndex["noDisk"] + (,$_)}

$ht.Keys | % {
$ht[$_] | % {
$curr = $_
if ($_.OSVirtualHardDisk) {$url = [uri]$_.OSVirtualHardDisk.MediaLink; $storageFileIndex[$url.Segments[-1]] = $storageFileIndex[$url.Segments[-1]] + (,$_)}
else {$storageFileIndex["noDrive"] = $storageFileIndex["noDrive"] + (,$_)}

if ($_.DataVirtualHardDisks) {
$_.DataVirtualHardDisks.DataVirtualHardDisk | % {
$url = [uri]$_.MediaLink; $storageFileIndex[$url.Segments[-1]] = $storageFileIndex[$url.Segments[-1]] + (,$curr)

What we need to do here is to run trough the array of VMs, extract a value of the property we are interested in and use it as a key in a hashtable. Value for this key is the object itself. For instance the simplest index would be an index by private IP address of the VMs. In this example we run through objects in the original hashtable, and for each of them extract IP address if the machine and create record in a new hashtable with this IP as a key and current object as the value.

$ht.Keys | % {
$ht[$_] | % {
if ($_.ipaddress) {$ipAddrIndex[$_.ipaddress] = $ipAddrIndex[$_.ipaddress] + (,$_)}
else {$ipAddrIndex["noip"] = $ipAddrIndex["noip"] + (,$_)}

At this point we have all our data in a hashtable and couple of indexes. In my case they are by IP address, by storage account and by vhd file names of the VMs. Using them we can search for a VM by using just a semantics of the hashtable like below. It returns all the VMs across all subscriptions having private IP address of


Next time we going to look at how to build a cmdlet that dynamically builds its parameters set based on names of indexes to do lookups on those indexes.

How to search for object in Azure

Problem statement

In our environment we have more than one subscription. Even more than two. And this amount grows constantly. What we do with all of this is we support VMs and stuff there. One of the issues is that users usually do not know name of Azure Service which is used to host their VMs, and it so happens that they do not know even their Subscription Name or Subscription ID. To bring up and control their VMs and environments the use some “middleware” which hides all of this info from them. So they usually come to us and say: “here you are a list of VMs we have problems with, please have a look and fix”. Sometimes such list contains not only VMs but storage accounts along with VMs etc. Unfortunately classic azure cmdlets does not provide an option to search for objects in the cloud by their name. On the other hand new Azure Management portal does. It can find objects by name. So I decided to do a little hack end use this API in order to simplify our live.

Continue reading How to search for object in Azure

Support Manifesto

Nowadays there is a big gap between developers and infrastructure operations. The problem is that they exist in two different worlds. They use different terminology and approaches, have distinctive strategies and goals, varying sets of knowledge, and even a different mindset. How can we work together like this, and who can join us?

We should all think not only in terms of our technical field, but also about the people we are doing this for. Customers don’t just require a solution itself. They also need to support and monitor this solution, manage it, integrate it with existing solutions and data flows, deploy new instances and restore failed ones. When a solution contains a number of components that can run on various servers and be moved from one to another, there is a lifesaving property that should be provided – the system should be self-descriptive. In other words, it should provide some interface to track its state and components in an easy way.

How can we achieve this? Here are some ideas and technologies to use for developers who want to do something really great.

Continue reading Support Manifesto

Bringing up Nano Server on Hyper-V

On the MS Ignite new thing, Nano Server was announced. The idea is to get smallest footprint for the Windows Server ever. Now it is possible to try it. In general it is quite straightforward. First you need to download the technical preview of new Server from here. “There you will find the gate to hell, opened before you. You must find the courage to step through that gate …” (c) oops ). No, not that bad.

After you have downloaded the image you can mount it and find a folder called NanoServer. There will be the wim image of the server and some additional folder with packages wich can be additionally added to the new installation. Next you need to follow the link here and read the documentation. What I did is just wrapped the doc into a small script which shows the steps in brief

$dism = "D:\temp\NewDism\dism.exe"
$imageFile = "D:\VMs\Virtual Hard Disks\nanoServerBaseLine2.vhd"
$mountDir = "D:\temp\mountdir"
$convert = "D:\Temp\Convert-WindowsImage.ps1"

& $convert -SourcePath E:\NanoServer\NanoServer.wim -VHDPath $imageFile -VHDFormat VHD -Edition 1
& $dism /Mount-Image /ImageFile:$imageFile /Index:1 /MountDir:$mountDir
& $dism /Add-Package /PackagePath:E:\NanoServer\Packages\ /Image:$mountdir

& $dism /Image:$mountdir /Apply-Unattend:D:\VMs\Unattend.xml

md $mountDir\windows\panther
copy D:\VMs\Unattend.xml $mountDir\Windows\panther

& $dism /Unmount-Image /MountDir:$mountdir /Commit

New-VM –Name testNanoVM –MemoryStartupBytes 1GB –VHDPath $imageFile -SwitchName Internal | Start-VM

How to bring up Windows 2012 R2 in HyperV without installing it

Hey, here is a great stuff out there. This script is able, so to speak, to convert an ISO file into VHD(x) file, that can be easily attached to the VM. Basically you take ISO, convert it, attach VHD to a Vm and voila, you are done. So what to do to achieve this? Easy, just download the script! And after that just:

.\Convert-WindowsImage.ps1 -source D:\X64FRE_SERVER_EVAL.ISO `
-Edition ServerDataCenterEval -VHDPath 'D:\VMs\Virtual Hard Disks\CONTOSO-DC1.vhdx' `
-VHDFormat vhdx -VHDPartitionStyle GPT


2015-05-08 18_41_49-Administrator_ Windows PowerShell

Quick and dirty way of checking links

So ho can we run through the site and check if links are ok on it? I mean if it is possible to automate this using powershell? Another thing here is how to do this fast, better in parallel threads. The problem here is that there is no threads in Powershell unfortunately. Thre are only Powershell Jobs, but they are quite heavy. So what to do and how to achive this? I’ve taken the idea from this post. In general i run Invoke-WebRequest cmdlet as a scriptblock in separate parallel runspaces. Here you can have a look. My contribution in this is not very big. I’ve just updated the scriptblock running as part of the thread like below:

$ScriptBlock = {
Param ($lnk)

$progressOld = $ProgressPreference
$obj = new-object psobject -prop @{Name = $lnk;
Status = (Invoke-WebRequest -Uri $lnk -DisableKeep).statuscode}

$ProgressPreference = $progressOld

Also i’ve added some checks to skip repeating links and links that does not start from “http*”