EMR digital ink handwriting recognition

Write notes in ink and convert to text

The “Star Trek” world where everyone can simply ask the computer a question or direct it to perform a task becomes closer every day. Thanks to Microsoft’s huge investment and research in human computer interaction we are closer to this reality every day. Medscribbler partners with Microsoft to not only lead in this technology for medicine but this is a case of medicine leading the world in new technology.

Digital ink and voice is a pervasive available input method in Medscribbler. Computer “inking” is really just one component of Microsoft’s initiative for intuitive human machine interaction that builds on the digital pen, digital ink, and speech technologies. With pen and speech input, wireless support, long battery life, portability and digital touch the evolutionary progress of computers is advancing quickly. Medscribbler is the advancement of this technology into a computer application.

Microsoft’s newest operating system now has the world’s most advanced system for the recognition of human generated communication, whether it is by touching, talking, messaging or the old “spelling it out” on a keyboard. The Microsoft operating systems allow application developers users to build software to can accept input with a digital pen as well as a standard keyboard, mouse or voice. Navigation and other functions are possible with touch. Other operating system vendors are hopelessly behind Microsoft’s development timetable. Hardware vendors are only beginning to exploit capabilities Microsoft has made possible. Medscribbler is already there with Microsoft. With capable hardware Medscribbler enables users to write or draw directly on the screen, navigate with touch and “talk” to their computer to not only generate a patient record but to process the information a medical provider needs to make decisions.

The computer handwriting process, called inking, enables users to add “digital ink” to Medscribbler, which appears as natural-looking handwriting on the screen. The digitized handwriting can be converted to standard text through handwriting recognition, or it can remain as handwritten text. Both the converted text in typeface and the cursive handwritten text function equally well as data formats in Medscribbler™ Both forms of text can be displayed from past notes, sent as a fax or converted to a PDF for export. Searching the text form or the handwriting form is also possible.

More Intuitive To Users

In many situations, handwriting is a more natural input method for placing data into applications. Inking makes Medscribbler, the first clinical processor, more convenient for users as both a medical encounter management tool and a producer of an electronic medical record. Standing beside an exam table and moving from room to room are medical situations where “fiddling” with a keyboard is challenging if not impossible. Medscribbler intelligently allows human computer interaction to replace the keyboard with “inking.”

Inking also allows users to insert a sketch or drawing, take free-hand notes, and annotate an existing document such as an X-ray. “Using a pen to input information into the Tablet PC is simply a more natural and productive way to work in many situations” says Microsoft® Corp’s Tablet PC expert Alex Gounares. “Using digital ink is so much quicker and easier than using a keyboard.”

“Thinking in ink”—maximizing the possibilities of ink as its own data format—is the other great potential of the Tablet PC and inking, according to Gounares. “There are so many cool things that we can do with ink as ink,” says Gounares.

Inking is actually a broad term that represents a set of technologies. A number of technologies had to come together to make inking possible. Many teams across Microsoft® —including Microsoft® Research teams on several continents—were involved in solving the technological challenges that came together as inking.

First, there were hardware considerations. The input screen required special features. On Tablet PCs and better touch screen monitors, active digitizers or passive touch screens, when the pen comes in contact with the screen’s electromagnetic field, its motion is reflected on the screen as a series of data points. As the pen continues to move across the screen, the digitizer or touch device collects information from the pen movement in a process called “sampling.” The sampling of “pen events” of motion correspond to data per second for processing by Microsoft’s operating system. These electromagnetic pen events are then represented visually on the screen as pen strokes.

Because of its high sampling rate and the operating system processing there is the effect of “real-time inking,” that is, as the user writes on the LCD screen, digital ink appears to flow at the same speed that the pen writes, no matter how fast the pen moves.

The high sampling rate also enables written ink to be displayed and stored with very high graphical resolution. Not only is this important for visual legibility on the screen, it is necessary for maximizing accuracy during the process of handwriting recognition. The more data points collected in the ink objects, the greater the accuracy when the data passes through the recognizer and is associated with words. This is why a digitizer is better than a lower technology touch screen.

“Microsoft is a leader in handwriting recognition,” says Gounares. “Microsoft has invested in this technology for many years, and we have built up considerable expertise and technology.” Former Microsoft chairman Bill Gates has revealed that the company devoted almost a billion dollars to the development of the recognition engine for both voice and handwriting.

The Microsoft operating system does not require the user to “train” the software to provide immediate functionality—the handwriting recognition engine can work right away, with any user. It is the problem of “universality” that other operating system and recognition software vendors have been unable to solve. “We have developed this technology by studying and analyzing large numbers of handwriting samples”, says Gounares. “Microsoft® has collected and stored such a large sample of handwriting in the recognizer that the recognition software requires no additional training… it is important to understand that handwriting recognition is inherently a statistical process.” For most users it works really well right away, for others, not so well right away. But beyond the “universality” right out of the box, users can customize and train the recognizer to their handwriting and voice. Medscribbler takes this a step further with context recognition in the software. Medscribbler “overrides” the general operating system for even better context sensitive results.

The inking process in Medscribbler gives users the choice of converting the handwritten or voice data to standard text through handwriting recognition. Users can also preserve the data in Medscribbler in its ink or voice format—keep ink as ink and voice as voice—and lose none of the data functionality. Hand-drawn or written ink and spoken voice need not be converted to a different format to be saved, sent, or otherwise manipulated by Medscribbler.

Handwriting preserved as ink objects are fully supported as a defined and recognized “native data type” across Windows® based applications and platforms. That is, digital ink can be passed from application to application as digital ink. For example, on a Tablet PC or touch monitor, users can respond to email in Outlook by writing responses in ink by hand onto the screen. The response is sent and received as handwritten digital ink—without the necessity of converting the handwritten ink objects to text.

This means that the Tablet PC is more than just a handwriting recognition tool. Notes taken in ink, sketches drawn in ink, annotations made in ink—none of these need to be converted to function as a data type in Medscribbler. A handwritten document has the same value and versatility as a document that has been keyed in on a keyboard.

Maintaining ink as ink is one of the most exciting breakthroughs Microsoft’s recognition engine. The beauty of ink in Medscribbler is that it allows you to express yourself freely. It preserves the context and personality of communication. You’re not restricted to any predefined structure: You don’t need to stay within the lines. Inking technology serves as the foundation for many innovations in Medscribbler that will join the naturalness of handwriting and drawing with the power of the PC.

“Over the course of the next few years, amazing applications will be developed for ink and voice” says Gounares. “Our mantra while developing the inking technology was that it had to be better than paper. We wanted the naturalness of paper with the power of the PC. There are few barriers to expressing yourself in ink.” Medscribbler is leading this application development.