In my post about having a standards headache, I asked how to handle getting invalid requests from your trading partners. I also touch on this issue in my book. I ask that just because you can fix something that your trading partner is doing wrong, should you. I want to dive into that question a little deeper with this post.
What is a schema?
First of all, let us make sure we understand what a schema is. A schema precisely defines the elements and attributes that are allowed in a data file. For our discussion, we will use XML (eXtensible Markup Language) to deliver and validate our data. So the schema tells us what is allowed in our data file, and if it deviates from that definition, it can cause the data file to be rejected.
I say that it can cause a data file to be rejected because no law says you have to validate your data file against a schema. You can just allow it to pass through into your backend process and carry on.
Not validating the data upfront is not a good idea because it will allow any data into your processing. Even data that could contain a nefarious payload in an attempt to “hack your system.”
Types of schema validation errors.
Something that doesn’t belong in the data file.
<UndefinedField>123</UndefinedField>
A field that has an incorrect data type. Something that should be numeric but has alpha characters, for example.
<TotalAmount>123.abc</TotalAmount>
A field could have a specific set of allowed values, and you could receive a value not in the allowed list.
<Gender>frog</Gender>
A field could be defined to not allow a null value, but a null value could be sent.
<NotNullableField></NotNullableField>
These are just a few of the types of errors you could see. There are countless others.
Where do you draw the line?
If one of your trading partners is sending you bad data files, what should you do about it? Do you “adjust,” or do you ask your trading partner to correct the data. I guess this is a rhetorical question because the answer is going to depend on your situation. But the one factor that will be common across all conditions will be money. So, yep, it is all about the Benjamin’s.
What does it cost your organization not to be able to process the files? How often is the error occurring? Can you just correct the data and reprocess it (should you even consider this)? How long will it take your trading partner to fix the issue (assuming they are willing to do so)? How long will it take you to “adjust” and allow the files to process normally?
In a perfect world.
You would always push back in a perfect world and ask your trading partner to fix the issue. If a schema exists, then there had to have been collaboration in creating it. So in effect, there is a contract between the two of you. And your trading partner is breaking that contract. Obviously, they are in the wrong.
Unfortunately, we don’t live in a perfect world. And to quote one of the great philosophers of all time, The Wicked Witch of the West, “Ohhhhh, what a world, what a world.”
What are your options?
Fixing the data.
Is this something you should do? For example, could there be legal implications if you alter the data you received directly from your trading partner?
If this is just a non-nullable field with a null value, and all you have to do is remove that field and re-drop the file, this could be okay. Especially if this is a “one-off” occurrence and you let your trading partner know about it. But if it is an everyday occurrence or multiple files per day, this is probably an untenable solution.
Asking your trading partner to fix the issue.
This is the most obvious and best solution for everyone. And I’m sure that in most situations they will agree to this. But it could be something that is not a priority for them, and it just goes into their backlog to be addressed sometime in the future.
Making an adjustment.
If you land on this option, the issue should have been looked at by several different people. This is not an ideal solution, but it may be the one that just makes the most sense. If making the change to your system, allowing the non-conforming data to process will wind up making money for your organization; this is the right thing.
It may cause you to throw up your hands and ask yourself why even bother to make data standards. But to quote another movie, The Godfather, “business is business, it’s not personal.”
I would love to hear how others have struggled/managed with this type of situation. Please leave a comment below.