[HAPI-devel] handling of HL7-null

[Life is hard, and then you die]

We're trying to handle hl7 update messages, and these may contain hl7
null's (""). HAPI does not currently seem to handle them specially,
which leads to various problems, such as validation failures and
making it somewhat torturous to detect these nulls.

I've been trying to figure out what the best approach for handling
nulls would be, and came up with the following: add two methods to
Type, 'boolean isNull()' and 'void setNull(boolean)'. The parsers
would set the null-flag appropriately, and the encoders and verifier
would simiarly check the flag; setting the flag on instances that are
part of another composite type (as opposed to part of the segment
directly) would have no effect.

I've done a quick implmentation of this and it works reasonably well.
But I wanted to gather feedback on the approach before submitting a
patch. Is there a better alternative? What have others done? Have I
missed something?

TIA.


Cheers,

Ronald

[Bryan Tripp]

Hi Ronald,

I'm not sure I understand the motivation for handling them specially.
Nulls (as opposed to empty fields) are to indicate that corresponding
data should be deleted from the receiving application. This is
supported -- if a field contains "\"\"" then the application should
delete data -- but it isn't HAPI's business to enforce this, so I
don't understand why special treatment is needed.

Can you explain further? E.g. what makes it difficult to check that
something is null, and what validation errors are you getting?

Bryan

Post by Life is hard, and then you die
We're trying to handle hl7 update messages, and these may contain hl7
null's (""). HAPI does not currently seem to handle them specially,
which leads to various problems, such as validation failures and
making it somewhat torturous to detect these nulls.
I've been trying to figure out what the best approach for handling
nulls would be, and came up with the following: add two methods to
Type, 'boolean isNull()' and 'void setNull(boolean)'. The parsers
would set the null-flag appropriately, and the encoders and verifier
would simiarly check the flag; setting the flag on instances that are
part of another composite type (as opposed to part of the segment
directly) would have no effect.
I've done a quick implmentation of this and it works reasonably well.
But I wanted to gather feedback on the approach before submitting a
patch. Is there a better alternative? What have others done? Have I
missed something?
TIA.
Cheers,
Ronald
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Hl7api-devel mailing list
https://lists.sourceforge.net/lists/listinfo/hl7api-devel

[Life is hard, and then you die]

Hi Bryan,

thanks for your reply.

Post by Bryan Tripp
I'm not sure I understand the motivation for handling them specially.
Nulls (as opposed to empty fields) are to indicate that corresponding
data should be deleted from the receiving application. This is
supported -- if a field contains "\"\"" then the application should
delete data -- but it isn't HAPI's business to enforce this, so I
don't understand why special treatment is needed.
Can you explain further? E.g. what makes it difficult to check that
something is null, and what validation errors are you getting?

There are three problems, really:

1. Checking for Null currently is hard, because you actually have
to check all parts in a composite type. E.g. for a field of type
XPN you have to check that XPN.1.1 is "", XPN.1.2 is empty,
XPN.1.3 is empty, etc, then XPN.2 is empty, XPN.3 is empty, etc.
Yes, if XPN.1.1 is "" then chances are small it wasn't an
Null, but, say, a surname of "" is legal if at least one other
component in the field is valued (i.e. a PID-5 field of ""^John is
legal, though unusual).

2. Profile validation fails when the first item in the type is
constrained to have a length of 1. E.g. Nulls in boolean fields
PID-8, PID-24, PID-30, PD1-12, and PV2-22 all cause validators to
complain.

3. I would argue that the current Null handling in the
RegexPrimitiveRule is wrong, because does not take the whole field
into account. For example a field of type DR with the value
2004^""^ (or e.g. a PID-5 of ^^^^^^^^^2004&"") passes validation
despite being invalid.

Having the parsers recognize Null and setting a flag instead of
assigning "" as the first component's value fixes all these problems.


Cheers,

Ronald

Post by Bryan Tripp

Post by Life is hard, and then you die
We're trying to handle hl7 update messages, and these may contain hl7
null's (""). HAPI does not currently seem to handle them specially,
which leads to various problems, such as validation failures and
making it somewhat torturous to detect these nulls.
I've been trying to figure out what the best approach for handling
nulls would be, and came up with the following: add two methods to
Type, 'boolean isNull()' and 'void setNull(boolean)'. The parsers
would set the null-flag appropriately, and the encoders and verifier
would simiarly check the flag; setting the flag on instances that are
part of another composite type (as opposed to part of the segment
directly) would have no effect.
I've done a quick implmentation of this and it works reasonably well.
But I wanted to gather feedback on the approach before submitting a
patch. Is there a better alternative? What have others done? Have I
missed something?
TIA.
Cheers,
Ronald

[Bryan Tripp]

Hi Ronald,

Thanks for the explanations. I think I understand where you're coming
from but I have a few more questions ...

Post by Life is hard, and then you die
1. Checking for Null currently is hard, because you actually have
to check all parts in a composite type. E.g. for a field of type
XPN you have to check that XPN.1.1 is "", XPN.1.2 is empty,
XPN.1.3 is empty, etc, then XPN.2 is empty, XPN.3 is empty, etc.
Yes, if XPN.1.1 is "" then chances are small it wasn't an
Null, but, say, a surname of "" is legal if at least one other
component in the field is valued (i.e. a PID-5 field of ""^John is
legal, though unusual).

I see what you mean technically, but I don't understand the intended
usage. Suppose you had a flag that told you something in a field was
null (without specifying which part). What would you do with this? Do
you have a routine that does nothing but handle nulls, and you only
want it to traverse only relevant parts of the message? So do you need
this flag at the segment level too?

Post by Life is hard, and then you die
2. Profile validation fails when the first item in the type is
constrained to have a length of 1. E.g. Nulls in boolean fields
PID-8, PID-24, PID-30, PD1-12, and PV2-22 all cause validators to
complain.

Good point, but can't this be fixed as easily without the flag?

Post by Life is hard, and then you die
3. I would argue that the current Null handling in the
RegexPrimitiveRule is wrong, because does not take the whole field
into account. For example a field of type DR with the value
2004^""^ (or e.g. a PID-5 of ^^^^^^^^^2004&"") passes validation
despite being invalid.

OK, but a PrimitiveTypeRule isn't supposed to be aware of rules that
span multiple components. The only support for this kind of thing
currently would be in ConformanceProfileRule. It would make sense to
add new functionality here with a CompositeTypeRule interface.
However, I don't see how the null flag would help with this. Can you
clarify?

Thanks,
Bryan

[Life is hard, and then you die]

Hi Bryan,

Post by Bryan Tripp
Thanks for the explanations. I think I understand where you're coming
from but I have a few more questions ...

Post by Life is hard, and then you die
1. Checking for Null currently is hard, because you actually have
to check all parts in a composite type. E.g. for a field of type
XPN you have to check that XPN.1.1 is "", XPN.1.2 is empty,
XPN.1.3 is empty, etc, then XPN.2 is empty, XPN.3 is empty, etc.
Yes, if XPN.1.1 is "" then chances are small it wasn't an
Null, but, say, a surname of "" is legal if at least one other
component in the field is valued (i.e. a PID-5 field of ""^John is
legal, though unusual).

I see what you mean technically, but I don't understand the intended
usage. Suppose you had a flag that told you something in a field was
null (without specifying which part). What would you do with this? Do
you have a routine that does nothing but handle nulls, and you only
want it to traverse only relevant parts of the message? So do you need
this flag at the segment level too?

I think there's a slight confusion regarding HL7 Null's: only a field
as a whole may be Null, not individual components. So, |""| says the
field is Null, but |^""| says the second component has a value of
"" (two quotes, not the empty string), i.e. in the latter case the ""
is not a Null. So the "" is not a flag that something in a field is
null, but a flag that the field itself, as a whole, is null.

[Bryan Tripp]

Hi Ronald,

Post by Life is hard, and then you die
I think there's a slight confusion regarding HL7 Null's: only a field
as a whole may be Null, not individual components. So, |""| says the

Yes, I might be confused about that. Can you give me a section reference?

Thanks,
Bryan

[Life is hard, and then you die]

[sorry for the slow reply - was offline for bit]

Post by Bryan Tripp
Hi Ronald,

Post by Life is hard, and then you die
I think there's a slight confusion regarding HL7 Null's: only a field
as a whole may be Null, not individual components. So, |""| says the

Yes, I might be confused about that. Can you give me a section reference?

Unfortunately the spec is not very explicit on this. However, there
are a couple implicit places (note that various other places in the
spec they talk about "null" when they actually mean "empty"):

- v2.5, Chapter 2, Section 2.5.3 says:

HL7 does not care how systems actually store data within an
application. When fields are transmitted, they are sent as
character strings. Except where noted, HL7 data fields may take
on the null value. Sending the null value, which is transmitted
as two double quote marks (""), is different from omitting an
optional data field. The difference appears when the contents of
a message will be used to update a record in a database rather
than create a new one. If no value is sent, (i.e., it is omitted)
the old value should remain unchanged. If the null value is
sent, the old value should be changed to null. For further
details, see Section 2.6, "Message construction rules".

(Section 2.6 however does not have anything further to say about
Nulls). Note that this is basically the only place that talks about
Nulls, apart from the update modes for repeating segments (2.10.4),
i.e. there's no mention anywhere that components or subcomponents
may be Null, only here that fields may be Null.

- v2.6, Jan/07 ballot spec, Chapter 2 (Control) is a bit more
explicit here (page 24):

A field may exist in one of three population states in an HL7 message

Populated. (Synonyms: valued, non-blank, not blank, not empty) [snip]

Not populated. (Synonyms: unpopulated, not valued, unvalued, blank,
empty, not present, missing.) [snip]

Null. Any existing value for the corresponding data base element
in the receiving application should be deleted. This is
symbolically communicated as two double-quotes between the
delimiters (i.e., |""|).Employing consecutive double quote
characters as the only content of a field for other purposes is
prohibited.

Further down on Page 56 (under repeating fields) it says:

With repeating fields, the segment action codes are not relevant.
Action codes cannot be applied to individual field repetitions,
because they cannot be uniquely identified. Therefore, they must
all be there. i.e., send a full list for each transaction. If the
intent is to delete an element, it is left off the list. This is
analogous to the snapshot mode for repeating segments and segment
groups. If the intent is to delete the whole list, the field is
transmitted once with a Null in the first component. In effect,
the Sender must make a statement about what action the receiver
is expected to take: omitting, or not populating, the field is
not a clear signal according to field state definition as
described in section 2.5.3.

At the same time, it is not incorrect to be precise about
specific information that is to be deleted if the data type
supports this capability. Note, however, that data types without
components, e.g. IS or ST do not support this capability. There
is no way to tie the Null value to an actual element instance in
the persistent data store. See example 2.

And in example 2 we find

The data type for PD1-1 is a data types without components. There
is no way to tie the Null value to an actual element instance in
the persistent data store. Therefore the following is ambiguous
and not good practice.

MSH|||||||||ADT^A31^ADT_A31|<cr>
EVN|...<cr>
PID|||1234567^^^^MRN|<cr>
PV1|...<cr>
PD1|S~""||||||||||||||||||||||


Cheers,

Ronald

[Bryan Tripp]

Hi Ronald,

Thanks very much for this helpful review.

I'm still not sure how to interpret this. The v2.6 text seems to
address "what can go in a field". If there's no corresponding text for
"what can go in a component/subcomponent" (is there? I don't have
access to this spec) then I think this would have to be interpreted as
an omission. Likewise the 2.5 text doesn't prohibit using nulls in
components, it just doesn't talk about the issue.

I were an application developer reading these specs, and I wanted to
tell other apps to delete an invalid subcomponent, I can imagine that
I would interpret the specs broadly and use |...""...|, rather than
sending two messages (with |""| and "...").

In the face of this ambiguity, and possible established practices, my
inclination as an API developer would be not to enforce the narrow
interpretation in this case.

James, do you have any comments?

Bryan

Post by Life is hard, and then you die
[sorry for the slow reply - was offline for bit]

Post by Bryan Tripp
Hi Ronald,

Post by Life is hard, and then you die
I think there's a slight confusion regarding HL7 Null's: only a field
as a whole may be Null, not individual components. So, |""| says the

Yes, I might be confused about that. Can you give me a section reference?

Unfortunately the spec is not very explicit on this. However, there
are a couple implicit places (note that various other places in the
HL7 does not care how systems actually store data within an
application. When fields are transmitted, they are sent as
character strings. Except where noted, HL7 data fields may take
on the null value. Sending the null value, which is transmitted
as two double quote marks (""), is different from omitting an
optional data field. The difference appears when the contents of
a message will be used to update a record in a database rather
than create a new one. If no value is sent, (i.e., it is omitted)
the old value should remain unchanged. If the null value is
sent, the old value should be changed to null. For further
details, see Section 2.6, "Message construction rules".
(Section 2.6 however does not have anything further to say about
Nulls). Note that this is basically the only place that talks about
Nulls, apart from the update modes for repeating segments (2.10.4),
i.e. there's no mention anywhere that components or subcomponents
may be Null, only here that fields may be Null.
- v2.6, Jan/07 ballot spec, Chapter 2 (Control) is a bit more
A field may exist in one of three population states in an HL7 message
Populated. (Synonyms: valued, non-blank, not blank, not empty) [snip]
Not populated. (Synonyms: unpopulated, not valued, unvalued, blank,
empty, not present, missing.) [snip]
Null. Any existing value for the corresponding data base element
in the receiving application should be deleted. This is
symbolically communicated as two double-quotes between the
delimiters (i.e., |""|).Employing consecutive double quote
characters as the only content of a field for other purposes is
prohibited.
With repeating fields, the segment action codes are not relevant.
Action codes cannot be applied to individual field repetitions,
because they cannot be uniquely identified. Therefore, they must
all be there. i.e., send a full list for each transaction. If the
intent is to delete an element, it is left off the list. This is
analogous to the snapshot mode for repeating segments and segment
groups. If the intent is to delete the whole list, the field is
transmitted once with a Null in the first component. In effect,
the Sender must make a statement about what action the receiver
is expected to take: omitting, or not populating, the field is
not a clear signal according to field state definition as
described in section 2.5.3.
At the same time, it is not incorrect to be precise about
specific information that is to be deleted if the data type
supports this capability. Note, however, that data types without
components, e.g. IS or ST do not support this capability. There
is no way to tie the Null value to an actual element instance in
the persistent data store. See example 2.
And in example 2 we find
The data type for PD1-1 is a data types without components. There
is no way to tie the Null value to an actual element instance in
the persistent data store. Therefore the following is ambiguous
and not good practice.
MSH|||||||||ADT^A31^ADT_A31|<cr>
EVN|...<cr>
PID|||1234567^^^^MRN|<cr>
PV1|...<cr>
PD1|S~""||||||||||||||||||||||
Cheers,
Ronald

[Bryan Tripp]

Hi Ronald,

Thanks for the explanations. I think I understand where you're coming
from but I have a few more questions ...

Post by Life is hard, and then you die
1. Checking for Null currently is hard, because you actually have
to check all parts in a composite type. E.g. for a field of type
XPN you have to check that XPN.1.1 is "", XPN.1.2 is empty,
XPN.1.3 is empty, etc, then XPN.2 is empty, XPN.3 is empty, etc.
Yes, if XPN.1.1 is "" then chances are small it wasn't an
Null, but, say, a surname of "" is legal if at least one other
component in the field is valued (i.e. a PID-5 field of ""^John is
legal, though unusual).

I see what you mean technically, but I don't understand the intended
usage. Suppose you had a flag that told you something in a field was
null (without specifying which part). What would you do with this? Do
you have a routine that does nothing but handle nulls, and you only
want it to traverse only relevant parts of the message? So do you need
this flag at the segment level too?

Post by Life is hard, and then you die
2. Profile validation fails when the first item in the type is
constrained to have a length of 1. E.g. Nulls in boolean fields
PID-8, PID-24, PID-30, PD1-12, and PV2-22 all cause validators to
complain.

Good point, but can't this be fixed as easily without the flag?

Post by Life is hard, and then you die
3. I would argue that the current Null handling in the
RegexPrimitiveRule is wrong, because does not take the whole field
into account. For example a field of type DR with the value
2004^""^ (or e.g. a PID-5 of ^^^^^^^^^2004&"") passes validation
despite being invalid.

OK, but a PrimitiveTypeRule isn't supposed to be aware of rules that
span multiple components. The only support for this kind of thing
currently would be in ConformanceProfileRule. It would make sense to
add new functionality here with a CompositeTypeRule interface.
However, I don't see how the null flag would help with this. Can you
clarify?

Thanks,
Bryan