XML Watch Folder Tutorial (Windows)

Automate your ingestion process by sending files to a 'watched' folder. This will then trigger a bespoke import process.

👨‍🏫Prerequisite knowledge: to get the best from this tutorial it will help to be familiar with scheduled tasks and editing PowerShell scripts

Contents

  1. Determine the trigger
  2. Configure the process
  3. Create the scheduled task
  4. Test the watch folder

 

  1. Determine the trigger

    In most cases, the watch folder process will be triggered by an XML sidecar file appearing in the watch folder, which will contain all metadata for the media. The metadata should contain an HTTP(s) link that can be accessed by our storage server IP address. 

    💡Pro tip: we strongly advise that you use XML files to trigger the process, as transfer times for media can cause bugs/delays

    Imagen storage service will need to retrieve the media during the ingestion process, so transferring media to trigger the process will duplicate efforts and significantly reduce efficiency  


    XML Watch folder (1)

     

    💡Pro tip: XML may be transferred via FTPS to trigger the ingest. Check out this guide for a walkthrough tutorial

    Configure the process

     

    The following code excerpt will construct the barebones of a Windows watch folder. Log files or transcripts can be added to aid debugging, and email functions can be added to alert users on failure. 

     

    #########################################
    ### Set the folder to monitor for XML ###
    #########################################
    $watchFolder = [location of your "Monitor" folder]

    ##################################################
    ### Set minimum TLS version required by Imagen ###
    ##################################################
    [Net.ServicePointManager]::SecurityProtocol =
    [Net.SecurityProtocolType]::Tls12

    ####################################
    ### SET TYPE OF FILES TO MONITOR ###
    ####################################
    $filterCreated = '*.xml'

    ###########################################################################
    ### CREATE FILE SYSTEM WATCHERS AND ONLY INCLUDE SPECIFIED FILE TYPE(S) ###
    ###########################################################################
      $fswCreated = New-Object IO.FileSystemWatcher $watchFolder, $filterCreated -Property @{
      IncludeSubdirectories = $false
      NotifyFilter = [IO.NotifyFilters]'FileName, LastWrite'
    }

    #########################################################
    ### GENERATE ACTIONS ACCORDING TO THE "Created" EVENT ###
    #########################################################
    $action=Register-ObjectEvent -InputObject $fswCreated -EventName Created -SourceIdentifier FileCreated -Action{

      #######################
      ### INPUT VARIABLES ###
      #######################
      $path = $Event.SourceEventArgs.FullPath
      $name = $Event.SourceEventArgs.Name  

        ######################    
      ### IMPORT PROCESS ###
      ######################
        Write-Host "Detected CREATION of $name" -ForegroundColor Green

        #################################
      ### SET HEADERS FOR REST CALL ###
      #################################
      $headers = @{"X-Imagen-API-Key"="$apiKey";"Authorization"="Bearer $bearer"}
      $contentType = "text/xml;charset=utf-8"

      #########################
        ### attempt REST call ###
      #########################
      try {
          ###############################
          ### REST CALL TO IMPORT XML ### 
          ###############################
          Invoke-RestMethod -uri "https://$mccHost`:83/imcc/v1/import/record/$mappingFile`?workflow=$videoWorkflow" -Headers $headers -ContentType $contentType -Method put -verbose -OutFile $mccResponseFile -InFile $path
          $RESTSuccess="true"
      }
      ###########################
      ### handle REST failure ###
      ###########################
      catch [Exception] {
          $restSuccess = "false"
          Write-Host "Exception = $_"

        ##########################################
      ### run final checks on import attempt ###
      ##########################################
      }finally {
          ##########################################
          ### check if REST call was successfull ###
          ##########################################
          if ($RESTSuccess -eq "true") {
      
              #####################################################
              ### Check the XML response to ensure valid import ###
              #####################################################
              $parseResopnse=[xml](Get-Content $mccResponseFile)

                ###############################
              ### set XPath to error node ###
              ###############################
              $failureError=$parseResopnse.Response.DatabaseJob.Error.InnerText
           
                    ##################################################
                  ### check to see if there is a failure message ###
                  ##################################################
                  if ($failureError -ne $null) {
                      Write-Error "Import failed because: $failureError"
                      $restSuccess = "false"
                  }#end if import failed

                   #######################################################################
                ### no error has been found so the import must have been successful ###
                #######################################################################
    else{
                    Write-Host "Successful Import -- $RESTSuccess"
                      $successIsoDate = Get-Date -Format s
                      $mccResponseContent=[xml](Get-Content $mccResponseFile)
                      $recordID=$mccResponseContent.Response.DatabaseJob.Import.ImagenRecords.ImagenRecord.ID
                      $mediaID=$mccResponseContent.Response.DatabaseJob.Import.ImagenRecords.ImagenRecord.MediaObjects.MediaObject.ID
                      write-host "Imagen Record Number: $recordID Media object number: $mediaID"
                  }#end else successfull import
          }#end if REST success
      }#end finally
    }#end action 

     

    💡Pro tip: the XML file dropped into the watch folder must be in an Imagen XML format that matched your record schema

     

  2. Create the scheduled task


  3. Test the watch folder

💡Pro Tip: Follow this link for more information on PowerShell execution policies


Alternative Considerations

Serverless code applications

Azure

You may have chosen to create - get from our PS team - an Azure function app, which will allow you to upload XML and media to an Azure storage account to trigger a function

Azure XML watch folder 


AWS

You may have chosen to create - or get from our PS team - an Azure function app, which will allow you to upload XML and media to an Azure storage account to trigger a function

AWS Lambda function