Being relatively new to SQL, this’ll be my first post for TSQL2sday. For those not in the know, TSQL2sday (or T-SQL Tuesday) was started by Adam Machanic (t/b) and, I quote from this month’s host Rob Sewell (t/b), is :
“…a chance for you to join in the SQL Server community and write a blog post on a suggested topic. It makes for a great way to find a bunch of blog posts showing the same subject from many different viewpoints”
This month’s topic (if you haven’t already guessed from the title) is Powershell.
Now, Powershell is my 1st coding love. I’m not brilliant at it, but I’m definitely more than a beginner. It’s what I spend most of my working life playing with, and it was the reason I eventually got in to trying to learn me some SQL. Baring all this in mind, this was the first month that I’ve felt confident enough to join in, so here goes!
Creating a Scheduled Powershell Task to Clean up Orphaned Files in our SQL Database.
Catchy title, right? Basically, what I have been playing with this month is using a combination of SQL and Powershell to help with a small mess that we have recently found ourselves in. For this to make sense, I guess it’s going to need some sort of context. I’ll try and be brief.
I work at a relatively small but successful law firm, and our practice pretty much lives and dies with our Case Management System (CMS). if you don’t know what that is then, suffice to say, it’s like an IT Service Desk tool on steroids, lots of steroids. This software stores all documents, letters, email and anything else that could possibly be associated with a legal case, and organises them all in some sort of logical fashion. Suffice to say we have literally millions of these files. One of the main ways that these files are imported into the CMS is by an Outlook plugin that files the email and / or attachments. The users is prompted to enter where they want to file them, and off they go.
We also use folder redirection for our users, so that their files are on a server rather than on their local computers. It works pretty well, has some issues here or there, but it’s OK. Folder Redirection isn’t the star of this show.
Over the last couple of months, there have been a couple of small issues with the server that houses the folder redirection data. This has seemingly had a knock on effect where emails filed into the CMS during a folder redirection blip are seemingly lost.
What’s the Problem?
Well, each file in the CMS has a database entry that points to the files location on a server somewhere. During a Folder Redirection blip, if you were to simply look at the CMS, it would appear that the files have been imported correctly. However, when trying to open them, you are greeted with “file can not be found”. What seems to have happened is that the file path has only the file name in it, not the actually crucial path part. After a lot of digging, these files (mostly) seem to have ended up in the root directory of the Folder Redirection share.
The first couple of times this happened, it took us ages to manually figure out where these were supposed to be filed, drag and drop the files there, and then manually updating the paths. The first time, this was irritating. The second time, infuriating. The third time lead me to this post, I wanted to never have to do this again!
There are a few things to note about this issue. Firstly, not all files I find in the database have been located in the Folder Redirection share root. Secondly, not all files in the Folder Redirection share root belong to Database entries with missing paths. Lastly, all files are associated with a legal case, these take the form of an entity (or “customer”) and a matter number. An example of this would be Entity ABC123 Matter 6 or ABC123/6. The files associated with this matter need to be stored in a path based on this, so for this matter the files would be stored in \\KSLFS01\ptrdata\docs\A\B\C\ABC123\6. To get each file going into the correct folder is something I needed to figure out.
This is my wonderful solution. Normally I’d use the code formatting of WordPress, but because its a Powersehll script with SQL code in there, it just didn’t format well, so excuse the picture :
This solution is essentially doing 3 separate functions combined together:
- Query the database to find all files in the database that meet my missing path criteria.
- Check the content of the Folder Redirection share root folder and check to see if the files are in there.
- If the files from the 1st step are found in the folder from the 2nd step, migrate the file to the correct location and update it’s path in the database.
SQL was obviously used to find the relevant files and then update the the file path when required. CONCAT and SUBSTRING were used to form the correct file path, and are things that I had not used before this,
What I like about this solution is that it is now something that can be scheduled weekly on the SQL server, and hopefully means that this problem should never really bother us again.
Now, neither the Powershell code, nor the SQL code, are all that complex here. This is the first time I’ve really tried to combine the two into some useful function. The Powershell itself is pretty straight forward but the SQL did take me a bit of research / trial and error to get right (the evolution can be found in this Github Gist, and this Twitter thread).
Whether what I have come up with is the best way to resolve this issue or not, is not important here. For me, this solution to me represents progress. For the last 10 months I have been teaching myself SQL, and I’ve been trying to learn Powershell for a good 4 years (but that last 2 seriously). This is the first time I am really seeing the fruits of my labour. The combination of using SQL and Powershell together made what would have been hours of tedious work into a relatively quick and painless process.
Going forward, I can see how combining SQL and Powershell could really open the doors to more solutions for problems I already have, problems I don’t know exist yet, or just improving some of my already existing solutions. I want to start using both of these languages to the full, and have already begun looking at resources, such as DBA Tools, (t/w) a great Powershell module for SQL admins, and have started reading some books by Itzik Ben-Gan (t/b), and working towards my MCSA. I’m really hoping that this is the beginning of something special!