Learning to Apply Cryptography

Working on the ProtectedData module has been quite a learning experience for me. The purpose of that module is to provide an API for PowerShell scripters which can be just as simple to use as the Data Protection API – via cmdlets such as ConvertFrom-SecureString and ConvertTo-SecureString – while allowing encrypted data to be securely shared among different users and computers. Incidentally, PowerShell is going to have built-in support for this kind of thing starting with version 5.0, via the new Protect-CmsMessage and Unprotect-CmsMessage commands, which you can see in the September preview of Windows Management Framework 5.0.

Both the ProtectedData module and the new CmsMessage cmdlets leverage public-key cryptography to securely share the data. Before working on this module, I had a fairly high-level, system administrator’s understanding of crypto. I knew how to deploy a public key infrastructure according to best practices, and how to request certificates from a public certificate authority, such as DigiCert. I knew the names of the commonly-used encryption protocols, which ones were for symmetric encryption, signing, or asymmetric encryption, and was reasonably up-to-date on the relative security of these algorithms with various key sizes. I knew that at a very high, conceptual level, public-key cryptography works through some sort of mathematical magic where you can take some sensitive data, apply an algorithm to it using one half of the public / private key pair, and then the only way to get the original data back is to apply an algorithm to the cipher text using the other half of the public / private key pair. The practical result is that so long as you know the private key is well protected, and you trust that the person holding the private key is who they say they are (via digital certificates), you can use this technique to protect to either ensure privacy, or to produce digital signatures.

In order to write the ProtectedData module, I had to deepen my understanding of these crypto algorithms a bit, both conceptually, and with regards to the specific libraries that I would be using. However, I don’t know how the actual math behind these encryption algorithms works, and I don’t much care, either. You don’t really need to know about the math unless you’re writing an implementation of an established algorithm, or you’re a hacker or cryptanalyst trying to crack them. All I’m concerned with is practical application: using established crypto libraries to protect data using various algorithms.

Here, I’ll share the general approach that the ProtectedData module uses for each type of encryption / decryption:


Symmetric Encryption

The module uses the AES algorithm with 256-bit keys for all of its symmetric encryption, but the process is largely the same no matter what algorithm you use. When you want to encrypt a piece of application data, whatever it happens to be, you use an algorithm such as AES to do that part. Symmetric algorithms are very fast, and they can be applied to data of any size. Generally speaking, you will use a randomly-generated key for each thing you want to encrypt. From that point on, everything becomes about securely sharing copies of that random, 32-byte AES key; you no longer have to worry about the application data itself, because everyone uses the same encrypted copy of that.


RSA

RSA is the most commonly-used form of public-key cryptography today, so far as I know, and it’s from RSA that my original understanding of public-key techniques originated. You only need a single public / private key pair in order to either sign data and verify signatures, or to encrypt and decrypt data. The basic process for sharing encrypted data looks like this:

  1. Encrypt your application data with AES, using a randomly-generated AES key.
  2. Using the recipient’s RSA public key, encrypt a copy of the AES key.
  3. Send the encrypted AES key to the recipient. Using their private key, they’ll be able to decrypt the AES key, and then use that to decrypt the actual data.

ECDH

Recently, I added support for ECC (Elliptic Curve Cryptography) to the ProtectedData module. ECC is a different mathematical approach to public-key crypto, which can supposedly be more secure than older algorithms such as RSA, while using much smaller key sizes.

There is no elliptic-curve variant of RSA, so in order to add some form of ECC support to the module, I had to figure out a new algorithm. In the .NET Framework, the only ECC algorithm that is suitable for protecting copies of AES keys is ECDH, or Elliptic Curve Diffie-Hellman.

Diffie-Hellman, unlike RSA, requires two public / private key pairs. At a very high level, the math magic behind Diffie-Hellman is based on the principle that if you combine the private key from one pair and the public key from another pair, using a certain algorithm, you will get the same result. It doesn’t matter which public / private key you use, so long as it’s one private and one public.

The overall process of using ECDH is a little bit more complicated than the process of using RSA:

  1. Encrypt your application data using AES, using a randomly-generated AES key. (I’ll refer to this as the “payload key” later, to avoid confusion.)
  2. Read the recipient’s ECDH public key from their certificate.
  3. Randomly generate a new ECDH public / private key pair. (This is generally referred to as an “ephemeral key”.)
  4. Using your new ephemeral private key, and the recipient’s public key, produce a second 32-byte AES key (which I’ll refer to as the “derived key”.)
  5. Encrypt your payload key with AES, using the derived key.
  6. Send the encrypted copy of the payload key, along with your ephemeral ECDH public key, to the recipient. They will be able to combine their ECDH private key with your ephemeral public key to produce the same derived key, which will decrypt the payload key, which will decrypt the data.

Password-derived keys

There’s a backup mechanism in the ProtectedData module which allows you to decrypt the data with a password, instead of with a certificate. Like ECDH, this involves producing a derived key, and then using AES to encrypt a copy of your payload key. So long as the receiving party knows the password, they can produce the same derived key.


That pretty much sums up the encryption / decryption techniques I’ve learned to leverage in scripts so far. This doesn’t touch on digital signatures at all; the ProtectedData module is purely for encryption and decryption at this point (to prevent information disclosure.) I may follow up on this article with a closer look at how each of these techniques is accomplished within the .NET Framework, if people are interested.

Posted in PowerShell, Professional | 9 Comments

Proxy Functions for Cmdlets with Dynamic Parameters

While working on an update for Pester today, I had to tackle an interesting problem. Pester’s mocking framework was not working well for commands that use dynamic parameters. What Pester essentially does, when you mock a function or Cmdlet, is create a new function with an identical param block to the one that’s being mocked.

When mocking an advanced function, this was a relatively easy problem to solve: just copy the original function’s DynamicParam block into the new one, and you’re done. However, I had to come up with another approach for mocking Cmdlets that have dynamic parameters. You might be thinking: “That’s nice. What does this have to do with Proxy Functions?” Well, when Pester mocks a Cmdlet, it’s creating a function with an identical param block. In that regard, it’s just like a proxy function, except it never calls the original Cmdlet.

The only hit I found on the internet when I searched for “Proxy Function” and “DynamicParam” was a Hey, Scripting Guy! guest article by Rohn Edwards. In that article, Rohn shows an example of creating a proxy function for Get-ChildItem, using an interesting approach of reading the output from Get-Command (with the -ArgumentList parameter) to find the dynamic parameters based on Path. I could probably have used that approach for every stock Provider-based Cmdlet, but it wouldn’t quite cut it for general use. Some Cmdlets might have dynamic parameter behavior that depends on parameters which aren’t passed by position, for example.

Instead, I decided to look for a way to get information about the dynamic parameters the same way the PowerShell engine does. Cmdlets which have dynamic parameters do so by implementing the IDynamicParameters interface, which has one method: GetDynamicParameters(). In theory, we should be able to create an instance of the Cmdlet’s type, set the properties associated with the static parameters, then call the GetDynamicParameters() method. With one small exception, that’s exactly how to get it working. The exception is that when you create an instance of a Cmdlet’s class, I couldn’t find a public way to assign its Execution Context, and that turned out to be necessary in order to call GetDynamicParameters() on many Cmdlets. I had to dip into Reflection slightly to assign a value to the Cmdlet’s private Context property:

function Get-DynamicParametersForCmdlet
{
    param (
        [string] $CmdletName,
        [hashtable] $Parameters
    )

    $command = Get-Command -Name $CmdletName -CommandType Cmdlet -ErrorAction Stop

    $cmdlet = New-Object $command.ImplementingType.FullName
    if ($cmdlet -isnot [System.Management.Automation.IDynamicParameters])
    {
        return
    }

    $flags = [System.Reflection.BindingFlags]'Instance, Nonpublic'
    $context = $ExecutionContext.GetType().GetField('_context', $flags).GetValue($ExecutionContext)
    [System.Management.Automation.Cmdlet].GetProperty('Context', $flags).SetValue($cmdlet, $context, $null)

    foreach ($keyValuePair in $Parameters.GetEnumerator())
    {
        $property = $cmdlet.GetType().GetProperty($keyValuePair.Key)
        if ($null -eq $property -or -not $property.CanWrite) { continue }

        $isParameter = [bool]($property.GetCustomAttributes([System.Management.Automation.ParameterAttribute], $true))
        if (-not $isParameter) { continue }

        $cmdlet.$($keyValuePair.Key) = $keyValuePair.Value
    }

    $cmdlet.GetDynamicParameters()
}

The version of this function in the Pester module has some extra error handling code to make it more robust; I trimmed down this snippet to make the basic points clear. Those are: Use Get-Command to find the name of the class that implements the Cmdlet. Use New-Object to create an instance of that class. Check to see if it implements IDynamicParameters. If so, set the Cmdlet object’s Context using Reflection, enumerate through the already-bound static parameters and set those properties on the Cmdlet object, then call GetDynamicParameters(). A call to this function is the only line in the DynamicParam block of the mock or Proxy function. For example:

DynamicParam
{
    Get-DynamicParametersForCmdlet -CmdletName Get-ChildItem -Parameters $PSBoundParameters
}
Posted in PowerShell | Tagged , , | 4 Comments

Example of dealing with (crappy) legacy code

This morning, I happened to be looking over some of my early PowerShell code, written about a year ago if I remember correctly. My first impulse was to eliminate all evidence that it ever existed. The problem is, I published it online, and tracking down every person who ever viewed or downloaded the thing seemed like a daunting task.

So, I decided instead to make a project out of cleaning up this old code, using the tools and skills I’ve picked up in the last year. In particular, I’ll be using Pester to write a suite of tests for the code, and then I’ll be refactoring it from a single 300-line Advanced Function into much smaller, easier to understand units.

I’m not sure how much time I’ll have to dedicate to this, and it will be fairly slow-going since I need to try to document my thoughts and changes every step of the way. However, when it’s done, I hope that it will present an interesting read for anyone who’s had to maintain legacy code (yours or someone else’s), opened up the file, and thought, “WTF?!”

At any point, you can refer to the repository’s log file to see the complete history of what’s been done so far, including links to the commits and files referenced in each step.

Posted in PowerShell | Tagged , , , | 2 Comments

2014 European PowerShell Summit

The European PowerShell Summit, organized by PowerShell.org, will be in Amsterdam September 29 – October 1 2014 at the Park Hotel. Details at http://powershell.org/wp/community-events/summit/

The Summit will feature 3 days of PowerShell sessions from PowerShell team members, PowerShell MVPs and other PowerShell experts. It’s the in-person gathering place for PowerShell enthusiasts and PowerShell users. It’s a place to make new connections, learn new techniques, and offer something to your peers and colleagues. If you can’t get your PowerShell questions answered at the PowerShell Summit you’ll never get an answer.

The Summit agenda is available to view at: http://eventmgr.azurewebsites.net/event/agenda/PSEU14

Registration is now open via http://eventmgr.azurewebsites.net/event/home/PSEU14.

Posted in PowerShell, Professional | Tagged , | Leave a comment

Code Coverage Analysis for Pester (Feedback Request)

(12-Jul-2014 Edit: This article is now quite a bit out of date. The Coverage Analysis feature has since been merged into Pester’s Beta branch, and some of the details of how it works have changed as well.)

I’m working on a new feature for Pester which provide a report on code coverage, in addition to what Pester already does when running the tests: https://github.com/dlwyatt/Pester/tree/CodeCoverageAnalysis

I’m looking for some feedback on how to build the user experience. It currently works by adding a single new parameter to Invoke-Pester (currently called -Coverage, but this may change). The Coverage parameter works a little bit like the -Property parameter on Select-Object, where you can either pass in a simple string value, or a hashtable which defines more options. Right now, the hashtable allows you to specify a specific range of lines within a file to analyze:

# Report on all of the ps1 / psm1 files in a folder
Invoke-Pester -Coverage .\*.ps1, .\*.psm1

# Report on lines 50 through 80 in a single file (perhaps if you want to
# show a coverage analysis for a single function)
Invoke-Pester -Coverage @{ Path = '.\MyModule.psm1'; StartLine = 50; EndLine = 80 }

# Combining strings and hashtables is fine
Invoke-Pester -Coverage @{ Path = '.\MyModule.psm1'; StartLine = 50; EndLine = 80 }, '.\MyOtherFile.ps1'

I plan to add support for analyzing functions by name, instead of having to use line number ranges, but right now I’m just curious about how user-friendly this approach is. If you think the current behavior is confusing, how would you improve it?

When you use this feature, it gives you a report containing the number of commands that were analyzed, the percentage of them that were executed by the test, and a table displaying the file path, line number and text of any commands that were missed. Right now this report goes to the console, but I’ll add it to the -Passthru output object at some point as well.

Posted in PowerShell | Tagged , , | Leave a comment

Musings on “Clean Code” in PowerShell

Lately I’ve been reading Uncle Bob Martin’s Clean Code. For those who haven’t read it, the book focuses primarily on the tasks that a developer should go through after “getting the code working.” Specifically, the steps required to convert the initial mess into something that is easy to read, understand and maintain. (This basically amounts to “have a complete suite of unit tests, keep them passing, and refactor until you’ve removed all the code smells.”)

The book uses Java in all of its code examples, but the guidelines he proposes are applicable to any object-oriented language. However, PowerShell isn’t really an OO language at this point. It’s a consumer of the .NET Framework, but support for defining new types is very limited. (You can either embed .NET code and compile it on the fly with Add-Type, or use some halfway measures to create dynamic PSObjects with properties / methods created via Add-Member.) For the most part, PowerShell itself is a procedural language.

On functions, the book offers a few main points of advice: They should be extremely short, should only have one reason to change (the Single Responsibility Principle), and the number of arguments should be kept to a minimum (zero, if possible). In the book, that last bit with zero arguments is made possible by Object-Oriented programming. You would “promote” some method arguments to be fields on a class instead, and just call the method on that class without having to pass in anything. DoSomething(param1, param2, param3) could be refactored into a Widget class with those three values as fields, where you wind up calling myWidget.DoSomething(); .

For years, I’ve avoided using global (or script) scoped variables wherever I could, believing it to be a bad practice. By having my functions accept all of their input in the form of parameters, and having them return output rather than modifying some global state, they become self-contained units of code which can be easily reused in other scripts on an as-needed basis. (Though this is mainly a concern from my VBScript days; in PowerShell, shared code gets placed into a module, rather than copied and pasted around.) The downside to this is a lot of extra “noise” in the script code; parameter blocks are partially repeated across many functions, and calls to those functions involve passing a large number of arguments (usually at least two, and frequently four or more.) Because I’ve also been in the habit of making every PowerShell function an Advanced function, the parameter blocks themselves can be quite long (3-4 lines per parameter, generally).

Based on the advice in Clean Code, I’m considering changing these habits for some of my PowerShell script code. By making use of the Script scope, a PowerShell script or module can emulate a “class” in some ways, removing the need to pass around a bunch of common objects as parameters. Whenever accessing these variables, you can prefix their names with $script: in order to make it clear that the variable is outside of the local scope. I could also drastically shorten the param blocks of some of my internal module functions. (Do I really need to specify things like [Parameter(Mandatory = $true)] all over the place, when I’m the one calling the functions to begin with, and there are unit tests in place to make sure I didn’t screw up?) These changes would go against all of the scripting habits I’ve built up over the past decade, but would certainly result in function definitions and calls that are much, much shorter than they are right now.

Here’s an example of an internal module function currently in the ProtectedData module, and how it might have its length reduced by 60 percent by simply getting rid of the bulk of the param block and rearranging the code slightly. The function exists at a point where $IterationCount has already been validated anyway. Likewise for the $Password and $Salt parameters; if they make it to the Get-KeyGenerator function, they’ve already been validated by the publicly exported module functions.

function Get-KeyGenerator
{
    [CmdletBinding(DefaultParameterSetName = 'CreateNew')]
    [OutputType([System.Security.Cryptography.Rfc2898DeriveBytes])]
    param (
        [Parameter(Mandatory = $true)]
        [System.Security.SecureString]
        $Password,

        [Parameter(Mandatory = $true, ParameterSetName = 'RestoreExisting')]
        [byte[]]
        $Salt,

        [ValidateRange(1,2147483647)]
        [int]
        $IterationCount = 1000
    )

    $byteArray = $null

    try
    {
        $byteArray = Convert-SecureStringToPinnedByteArray -SecureString $Password

        if ($PSCmdlet.ParameterSetName -eq 'RestoreExisting')
        {
            $saltBytes = $Salt
        }
        else
        {
            $saltBytes = Get-RandomBytes -Count 32
        }

        New-Object System.Security.Cryptography.Rfc2898DeriveBytes($byteArray, $saltBytes, $IterationCount)
    }
    finally
    {
        if ($byteArray -is [IDisposable]) { $byteArray.Dispose() }
    }

} # function Get-KeyGenerator
function Get-KeyGenerator([System.Security.SecureString] $Password,  [byte[]] $Salt,
                          [int] $IterationCount = 1000)
{
    $byteArray = $null
    try
    {
        if ($null -eq $Salt) { $Salt = Get-RandomBytes -Count 32 }
        $byteArray = Convert-SecureStringToPinnedByteArray -SecureString $Password
        New-Object System.Security.Cryptography.Rfc2898DeriveBytes($byteArray, $Salt, $IterationCount)
    }
    finally
    {
        if ($byteArray -is [IDisposable]) { $byteArray.Dispose() }
    }
}
Posted in PowerShell, Professional | 3 Comments

Thoughts on self-defense and “Stand Your Ground”

This post is a follow-up to a brief conversation that took place on Twitter. 140-character chunks are really not a suitable venue for such a touchy topic.

Before I go into my take on this topic, let me put out the standard disclaimer: I am not a lawyer, a cop, or any sort of legal expert. However, for several years when I lived in Ohio, I did have a Concealed Handgun License, and have spent many, many hours on classes, training, and personal reflection regarding different situations where I might use (or not use) deadly force, and whether it would have been legally justifiable as self-defense. My statements in this post will be based on Ohio law that was current as of around 2008. Different State laws may have different details, even if the overall picture is fairly close to what Ohio had. My memory of the details might also be a bit hazy, since I’ve been living in Canada for quite a while now, and no longer own any guns.

First, the legal stuff. If you use deadly force, you’re almost certainly going to face criminal charges and wind up on trial. Self-defense is an “affirmative defense”, which carries a nasty meaning: Instead of “innocent until proven guilty”, you’ve just confessed to committing murder (and entered a Guilty plea, if I remember correctly), and the burden of proof is on YOU to convince a jury that it was justified as self-defense. Reasonable doubt is now against you, and if you fail, there’s no second chance (other than through the appeals process.)

There are several requirements for a self-defense claim:

  • You must not have helped to create or escalate the situation that led to the use of deadly force. If, for example, you get into an argument, then it moves to pushing, fists, and so on… back the hell off. Even if the other person introduces deadly force first, and you’re technically defending yourself at that point, you’re still guilty, legally speaking. I have no idea how George Zimmerman got off in the face of this point, but maybe Florida’s laws were different. There was a ton of evidence showing that the police told him to back off, and he pursued Trayvon anyway. George should be in prison right now, based on that alone.
  • You must be in reasonable fear of death or serious harm (rape, critical injury, etc.) You don’t have to wait for someone to shoot first, so to speak, but you can’t just employ deadly force against someone who is yelling at you, either. The “reasonable” part is tricky, because it’s different for every person, taking into account things like disparity of size and strength between the attacker and defender, number of people involved, how things escalated, etc.
  • You cannot have been commiting a crime at the time that the situation broke out. If you’re in an illegal gambling house, dealing drugs, breaking and entering, or whatever, you no longer have a right to a legal self-defense claim. Even if you are forced to defend yourself with deadly force, you’re probably going to be convicted as a result.
  • Duty to retreat. Here’s where the various “Stand your Ground” or “Castle Doctrine” laws come into play; they remove the duty to retreat in some or all situations. However, where duty to retreat still applies, you have to make an effort to remove yourself from the area before employing deadly force. You’re only justified in a self-defense claim if you tried to leave, and the attacker pursued you anyway. In a state with full “Stand your Ground” laws, this requirement doesn’t exist. Anywhere you have a legal right to be, you also have a legal right to defend yourself IF all of the other conditions apply. That’s not to say that you shouldn’t try to remove yourself from the situation, if you can safely do so, but it’s no longer a legal requirement. If a situation develops / escalates quickly, you may not have an opportunity to leave without putting yourself in further danger, and that’s the reasoning behind removing the duty to retreat.

It’s also permissible to act in defense of another person, so long as they would have been justified in using deadly force themselves, according to all of these same requirements. However, where strangers are involved, you have no way of knowing whether they escalated the situation or were committing a crime, which makes it very risky (legally) to intervene in an attack on someone else.

Because a jury trial is involved, many of these points are subjective. Depending on what other evidence is available, it may literally come down to how convincing your story is, and whether the jury believes you. That’s scary as hell, and should make anyone think twice before they start shooting. Don’t overreact to mild, potential threats. Think through scenarios in advance to train yourself to react correctly when the adrenaline hits. Practice your awareness and avoidance so you have plenty of time to see a potential threat coming. Don’t put yourself in dangerous positions in the first place, if they can be at all avoided. Be noisy as hell; attract attention before the situation explodes, if you can. If the attacker continues to pursue you, at least there will be witnesses to help back up your story later. None of this changes, regardless of whether guns are involved.

There is one point legal point which I feel is ridiculous in some states, Florida included, and that’s an “immunity from prosecution” clause. Zimmerman was originally not even arrested after his shooting because of such a law. It’s a chicken or egg problem, though: If you use legally justified deadly force, you can’t be put on trial. How the hell do you prove legal justification without a trial? Everyone should have to prove self-defense in a courtroom, if they’ve used deadly force on a person (whether it’s a shooting, or any other type of situation.)

I’ve seen some people make glib comments like “I was afraid, so I stood my ground and started shooting”, which makes light of a very serious topic. Regardless of your feelings about guns and their owners, deadly force is no joking matter.

Posted in Personal | Leave a comment

How to install a DSC Pull Server on Windows 2008 R2

When you set up a DSC pull server on Windows Server 2012 R2, you can simply run the command (Add-WindowsFeature DSC-Service) to install all of the necessary dependencies: IIS, ASP.NET, OData, and so on.

It appears to be possible to use older operating systems as DSC pull servers as well (so long as WMF 4.0 is supported on that platform), but you have to set up all of these prerequisites yourself. Here’s the process I used to build a Windows Server 2008 R2 DSC pull server:

– Install Windows Server 2008 R2 SP1
– Install Microsoft .NET Framework 4.5.
– Install Windows Management Framework 4.0. (reboot required after this step)
– Install IIS. Select all of the default options, plus “Tracing”.
– Run dism /online /Enable-Feature /FeatureName:ManagementOdata
– Run dism /online /Enable-Feature /FeatureName:DSC-Service
– Run $env:windir\Microsoft.NET\Framework\v4.0.30319\aspnet_regiis.exe -i

The rest of the process is common to all versions, setting up the website and application pool in IIS. This process is detailed at http://powershell.org/wp/2013/10/03/building-a-desired-state-configuration-pull-server/ , and there are several examples of scripts to perform these steps out on the internet as well (though the scripts may not have been tested on IIS7, so be careful there.)

Posted in PowerShell | Tagged , | 11 Comments

Quick tip on XML / XPath filtering of Event Logs

I’ve seen several examples of XML / XPath filtering of event logs, some of them from Microsoft, which look like this:

*[EventData[Data[@Name='SubjectUserName'] and (Data='test9')]] 

This sort of works, but you need to understand what it really means. It looks for a Data node under EventData which has a Name attribute of SubjectUserName, then it looks for a Data node under EventData whose text data is equal to ‘test9’. What’s important here is that the Data nodes identified by these two conditions don’t actually have to be the same. Most of the time, this won’t be a problem, but let’s assume that you want to look for events where the SubjectUserName is NOT equal to test9:

*[EventData[Data[@Name='SubjectUserName'] and (Data!='test9')]] 

This command would most likely return all events with a SubjectUserName data field, even the ones that were equal to ‘test9’. Again, breaking down the logic here, you’re saying “Look for events that have a Data node named SubjectUserName, and a Data node whose value is not equal to test9.” It’s not actually saying “look for Data nodes named SubjectUserName that are also not equal to ‘test9′”, which is what the intent here should be. To fix these XPath queries, do this instead:

*[EventData[Data[@Name='SubjectUserName'] != 'test9']] 

This can be useful in PowerShell if you want to use Get-WinEvent’s FilterXPath or FilterXML parameters, and also just for defining filters and custom views in the Event Viewer (including specifying which events to include in Event Forwarding.) They all use the same XML / XPath syntax.

Posted in PowerShell | Tagged , , | 2 Comments

Tracking down commands that are polluting your pipeline

This is a PowerShell “gotcha” that has probably caught every one of us at some point or another: You are writing a function… it’s intended to output a certain type of object, perhaps even a single value such as $true or $false in a Test- function, and you run it. You get a bunch of random garbage output, because some of the commands in your function are generating output that you forgot to suppress (or just didn’t know about). For example:

function Test-Function
{
    $arrayList = New-Object System.Collections.ArrayList

    $null = $arrayList.Add('One')
    [void] $arrayList.Add('Two')
    $arrayList.Add('Three') | Out-Null
    
    $arrayList.Add('Four')

    return $true
}

I’ll pick on ArrayList today, since its Add method returns an integer result which you may or may not actually care about. I suppressed the output from the first three calls to $arrayList.Add() using three different methods: assigning the result to a variable (or to $null if you just want to discard it), casting the result to [void], and piping the result to Out-Null. All three are valid, though the pipe to Out-Null is noticeably slower, if you’re doing this in a loop a few thousand times.

The fourth call, however, produces an integer result which is allowed to be written to the Output stream of the function. If you run this function as-is, it will write both “3” and $true to the output stream, instead of just the $true value you had intended.

Now, this is a small function, where it’s easy to spot the error. Ideally, all functions would look like this: neat and tidy, easy to debug. Unfortunately, this is often not the case. You may have written a function that grew quite large and hasn’t been refactored yet, or you may be inheriting and debugging someone else’s code. In the past, I’ve given the code a look over first, to see if I can spot the problem. If not, I would step through it in the ISE, hitting F11 until something unexpected showed up in the console window.

Today, someone asked about this topic on the TechNet forums, and for whatever reason, said debugging the script wasn’t going to be a viable option for him, so I started looking for a quicker, automated way to get this debugging done, and found that by piping the output of the command you want to debug into a function which looks at Get-PSCallStack, we can identify the culprits pretty quick (at least on PowerShell 3.0 or later; the Get-PSCallStack cmdlet doesn’t provide quite a useful information in version 2.0.) I tweaked the test function slightly to call on both another function, and another PowerShell script, and piped the output to this Get-OutputProducingCommand filter:

filter Get-OutputProducingCommand
{
    $output = New-Object System.Text.StringBuilder

    $stack = @(Get-PSCallStack)
    $indent = 0

    # Ignoring the top level or two of the call stack, where we've invoked the script.
    # This behavior is slightly different depending on whether the script was run at
    # a prompt, or executed by the ISE with F5.  We also ignore the stack entry at index
    # 0, which will always be the Get-OutputProducingCommand filter itself.

    $startIndex = $stack.Count - 1
    if (('<No file>', 'prompt') -contains $stack[$startIndex].Location) { $startIndex-- }

    for ($i = $startIndex - 1; $i -gt 0; $i--)
    {
        $null = $output.AppendFormat("{0,$indent}{1} ({2} Line {3}): {4}`r`n",
                                     [string]::Empty,
                                     $stack[$i].Command,
                                     (Split-Path -Path $stack[$i].ScriptName -Leaf),
                                     $stack[$i].ScriptLineNumber,
                                     $stack[$i].Position.Text)          
        $indent += 2
    }

    $output.ToString()
}

function Test-Function
{
    $arrayList = New-Object System.Collections.ArrayList

    $null = $arrayList.Add('One')
    [void] $arrayList.Add('Two')
    $arrayList.Add('Three') | Out-Null
    
    $arrayList.Add('Four')

    Test-NestedFunction

    # testNestedScript.ps1 is just a one-liner that returns a string,
    # to make sure this code displays useful output when calling into
    # an external script file instead of just calling a function.
    
    .\testNestedScript.ps1

    return $true
}

function Test-NestedFunction
{
    Write-Output 5
}

Test-Function |
Get-OutputProducingCommand

By piping our offending function Test-Function to Get-OutputProducingCommand, we get the following result. It’s a fairly quick and painless way to narrow down the search for those unruly commands that are sending junk down the pipeline.

Test-Function (Test.ps1 Line 9): $arrayList.Add('Four')

Test-Function (Test.ps1 Line 11): Test-NestedFunction
  Test-NestedFunction (Test.ps1 Line 20): Write-Output 5

Test-Function (Test.ps1 Line 13): .\testNestedScript.ps1
  testNestedScript.ps1 (testNestedScript.ps1 Line 1): 'This is a test!'

Test-Function (Test.ps1 Line 15): $true
Posted in PowerShell | 1 Comment